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A Message from the Organisers 


AUUG Inc is proud to present its 1993 Summer Conference to Melbourne’s open 
systems community. 

A quick glance through this year’s programme shows a surprising spectrum of 
topics. This is a direct reflection of the explosive growth of Unix (and, indeed, open 
systems in general) over the past few years. Despite this rapidly broadening context, 
I think you’ll all agree that we have retained the technical nature of the event. 

The programme at hand is one of the finest I’ve seen assembled for a summer 
conference, and this is a direct credit to the many gifted speakers you’ll hear today. 
Volunteering to present a paper is a heavy commitment, but it is the most effective 
way to contribute towards the local Unix community. My thanks to all our speakers. 

Of course, a conference cannot be organised in a day and there are certainly some 
people without whom it would not have been possible at all. Jon Eaves and John 
Carey helped me when I was snowed under and hounded me when I was forgetful. 
Also invaluable was the assistance of Steven Lynch and Chris Karadaglis, envelope 
stuffers extraordinaire. 

Silicon Graphics generously sponsored the printing of these proceedings, and thus 
providing a pennanent record of the conference. 

Lastly, AUUG deserves recognition for providing encouragement and financial 
support for this conference. Especial thanks are due to Glenn Huxtable and Liz 
Fraumann. 


Michael Paddon 
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Stockbroking with Open Systems 


Gregory Bond, 

Burdett Buckeridge & Young Ltd. 
gnbQbby.com.au 

February 14, 1993. 


Abstract 

I begin with an examination of why a Stockbroker would choose an Open Systems 
solution. I then trace the history of the Unix 1 -based dealing systems over the five 
year history of the firm, looking at the choices made at each stage in the developing 
technology. Special attention is paid to the decision to choose X-terminals rather than 
workstations as the next evolutionary step; how we implemented this choice; and the 
results of the project. Finally, I will present a technology wish-list to vendors who 
would like to sell into sites similar to ours. 


1 Introduction 

Burdett, Buckeridge & Young (BB Y) is a small institutional stockbroker. Since the company 
was founded in 1988 there has been a commitment to state-of-the-art information technology 
(based on Open Systems workstations) to provide a competitive edge to the dealing staff. 
This paper examines some of our experiences in applying this technology to the dealing 
information systems. Much of the rest of the office use PCs; that story (and integrating 
them with the main Unix system) is challenge for another paper and not in any way unique 
to a stockbroking business. 

1.1 What does a Stockbroker do? 

A stockbroker buys and sells shares and derivatives in listed companies on behalf of clients. 
At BBY these clients arc large institutional investors, such as insurance companies and 
superannuation funds. This section of the stockbroking business is very competitive, with 
many brokers chasing the business of a fairly small number of clients. Most clients have a 
panel system that regularly rates brokers on a combination of service factors. The brokers 
that come out well in the panel reviews get a large proportion of the business from that 
client. Hence the range of services provided (dealers advice, research analysis of companies, 
efficient processing of transactions, expert advice and reports etc) is the “product” that a 
broker sells. It is also a highly leveraged business, so a small edge in service can bring a 
large increase in income if you can climb one position on a panel ladder. Like all service 
industries the cost base is primarily in staff, which means maximising staff productivity is 
a major goal. 

The word “stockmarket” usually brings up two images. The first is of a pit full of 
men (almost entirely men!) shouting at the “chalkies” who write the prices on the big 
blackboard. The second image is of brokers jumping out of windows when the market 
crashes (as it will almost inevitably do from time to time). As to the first, since 1990 this 

1 Unix is a bell of AT&T Trademark Laboratories. 


1 







is no longer how the Australian market operates 2 . These days, the market is done entirely 
electronically via a system called SEAT (Stock Exchange Automated Trading). This is 
built using a network of slightly customised 286 PCs on brokers’ desks to enter orders and 
watch the market, driven by a bunch of Vaxen. As to the second image, our office is on the 
second floor so broken bones are probably all that would happen! 

1.2 Typical Information Sources 

The information from the SEAT system is transmitted in real time to brokers and clients. 
It is this data that forms the heart of a dealer information system. Presenting this data in 
a way that is accessible and current (usually within a second of the change in the market) 
is the main challenge. There are a number of other sources of relevant data: news reports 
containing headlines and articles of financial news (especially vital when economic statistics 
are released, when a delay of 10 seconds can mean fortunes made and lost on the futures 
market); announcements and statutory reports from various companies; and information 
about companies and shares (dividends, name changes, number of shares, etc.) These data 
feeds come from a variety of sources (the Stock Exchange is the main source; third-party 
vendors provide various additional feeds) and in a variety of formats (free text, COBOL-like 
records, binary records) and a variety of distribution media (leased line, X.25, microwave, 
ethernet/NETB ios). 

Added to these “primary” information sources are an open-ended range of possible 
secondary sources, which take the primary information and merge, store, manipulate and 
display useful histories or summaries. The most obvious step is to store daily data in 
a historical database, so that prices (or whatever) can be compared over time, usually 
graphically. Other common reports include top n stocks by traded value, index weightings 
(a measure of how “important” a stock is by measuring the value of all its shares) and 
dividend yield. More esoteric (and valuable) reports are aimed at helping clients maximise 
the return on their investment by using various derivatives (options and futures) in a variety 
of strategies. 

The challenge for the broking EDP team is to present this overwhelming mass of data to 
the dealers in a way that is consistent, useful and relevant, bearing in mind that the dealers 
are in the main financial people rather than computer people, generally unfamiliar with 
computers (and often intimidated). It is obvious that this information system is mission- 
critical! Without it available we can do no business, and the standing of the firm in the 
eyes of the clients is diminished. There are a number of companies who specialize in 
providing this integration, and produce systems that will take all the data feeds and put a 
single integrated dealing system on each desk. 

1.3 Typical Broker Systems 

The usual result of the plethora of data sources has been that each dealer station has a 
large number (up to five or more) terminals, one from each of the information vendors, 
each providing access to a limited range of features. The drawbacks of this approach are 
obvious; each terminal has a different, incompatible and often impossibly obscure interface. 

One popular solution to at least the desk crowding problem is video switching. Each 
desk has a single keyboard and display, but they can be switched at will to any one of the 
vendor screens. This works tolerably well for read-only sources (such as news or company 
reports), and has a degree of vendor-independence. However, you still haven't unified the 
interface. 

A solution that has come of age in the last two years is a PC/Windows based system. 
Here each desk has a single PC running a single integrated application,communicating via 

2 Allhough you slill see footage of this on the news every time there is a story on the market, including one clip 
shown regularly on ABC that contains an ex-employee of BBY who left the industry in 1989. 
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a PC network to a server that contains interfaces to the various data sources. This fixes the 
two main problems, but is still less than ideal in a couple of areas. The main shortcoming is 
the lack of resolution of a VGA screen, and the consequent limit on the amount of data on a 
screen at once. A second problem is the poor reliability of PC software and (especially) PC 
networks. The computing power of a PC may also be insufficient for some applications. 
There are about three vendors of such systems, one with a very large share of the market 
and two newcomers. Had such systems been available five years ago they may well have 
been the solution BBY chose. 

2 Why did BBY choose Open Systems? 

When the company was started in early 1988 one of the major decisions that had to be made 
was to choose a dealer information system. We could have gone with one of the existing 
integrated vendors, but the solutions were less than ideal. An additional factor was the “me, 
too” effect. As a newcomer the firm needed to differentiate itself from the pack, and using 
the same information systems as the rest of the brokers would not help that. 

Starting from scratch was a big advantage in that we had no investment in hardware, 
software or user skills, so we were free to chose a novel solution. One of the principals 
of the firm was a physicist in a previous life and had had contact with workstations as a 
postgrad student. From a hardware viewpoint, these were the ideal machines: a single, 
large, hi-res color screen, one keyboard, one mouse, one interface. 

That left the question of software. In the U.S., workstations were starting to appear 
on Wall St., and there was software available for U.S. markets. This was not suitable 
for local conditions (as Australian markets are substantially different to U.S. markets 3 .) 
However, a small Sydney software house (now deceased?) had developed software to do 
the main real-time handling of price and trade data. This, with the purchase of an RDBMS 
(Ingres), formed the basis of the dealer system; putting the rest together was thought to be 
a “Simple Matter Of Programming”. One person was employed in the EDP “department”, 
with responsibility for the whole EDP system including all development and support for 
the PC users. 

3 A History of the BBY Installation. 

The initial purchase was was a Sun 3/260 server in Melbourne to run the main quote software 
and the RDBMS, a headless 3/60 server in Sydney to run the quotes there, a leased line 
to connect the two cities, and two 3/60 workstations in each city as dealer screens. Other 
dealers used ASCII terminals and a hacked-on version of the screen program to provide 
access to multiple pages of data. 

The company began trading on 1 July 1988 with a very bare-bones dealing system 
consisting of an interim version of the quote software and little else. (Even this small 
facility was not substantially less than a lot of other brokers had access to). Development 
was happening at a frantic pace (largely with contractors) to provide all the necessary 
functions, and a lot of “80-20” solutions were implemented. Very shortly it was discovered 
that one person was not nearly enough and I was hired, to be followed by two more 
programmers as the size of the task grew apparent. 

The initial windowing environment was of course SunView. The Ingres 4GL was used 
for the database programs, and the “look & feel” of the 4GL programs (mainly in the use of 
function keys and menu lines) was copied in the non-DBMS programs (as far as possible). 
Because only a few of the dealers had workstations, and various PC users around the office 

3 At least here we quote things in cents. The U.S. market quotes tilings in l/8ths (and smaller power-of-two 
fractions) of dollars, a historical carryover from the “pieces of eight” that pirates used to plunder! 
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needed access to the same programs, the initial work was entirely on text-based applications 
(a situation that remains largely true: probably 90% of all the applications are text-only.) 

Right from the early days, the flexibility of rolling your own software was demonstrated. 
For example, using the publically-available sc spreadsheet as a basis, we had a customised 
spreadsheet program, with built-in financial and option pricing functions and live real-time 
access to price data available within a few months. This was revolutionary at the time and 
not available to any other broker; as a result a number of special deals were constructed and 
executed for clients. As part of this, we purchased the source rights to the quote system 
software to allow us to add our own functions. The authors of this package subsequently 
dropped it and we remain (as far as I know) the only users of it, although it has changed 
enough over the years that the authors wouldn’t recognise it. 

It was clear at this point that we had a “distributed" environment, thus conforming to 
the buzzword of the month. 

3.1 Open Systems? Open Software? 

The workstations proved to be popular (and something of a status symbol, there being fewer 
workstations than dealers), so the number of workstations increased. As can be expected, 
we rapidly ran out of puff on the server as more and more applications were developed and 
implemented. One application in particular was severely limited by the DBMS speed. Tins 
led us to investigate a new server machine to run the DBMS. 

It was clear at this time (mid 1989) that the Sun 3 architecture had come to an end, so we 
were more or less forced into changing architectures. However by this stage the investment 
in software (in particular the DBMS applications and the not inconsiderable investment in 
the DBMS software itself) meant that we had no real alternative but to purchase equipment 
that would run Ingres. Open Systems meant on one hand taht we had a wide choice of 
hardware vendors; on the other hand we were locked in to our software. There is probably 
a general lesson in this: Choosing software is much more important in the initial stages of 
implementing an Open Systems solution than choosing hardware. (I should point out that 
we were and are happy with Ingres, so this was not a burden to us.) 

We investigated a number of hardware alternatives: Pyramid was too big and expensive; 
Sequent was too big and aimed more for lots of little jobs rather than the one big job model 
that our usage represented. The short list came down to MIPS and SPARC, both from Sun 
and Solbourne. In the end, we bought a Solbourne (one of the first in Australia, serial 
number 101) on the basis of bang for buck (about twice that of Sun at that time, and about 
50% better than MIPS). Although Ingres was not at that time certified on Solbournes, we just 
loaded the tape and it ran. Converting our applications was simple but not automatic. Most 
of the changes were because the SPARC ran SunOs 4.0 rather than 3.5 we were running on 
the Sun 3s. Maintaining applications on two hardware architectures was a bit of a hassle, 
but with some discipline on the part of the system administrator and the development team 
it was easily manageable. Maintaining two different operating systems (3.5 and 4.0) was 
more of a challenge for the administrator (me); that problem was greatly relieved when we 
upgraded our Sun3s to the 4.0 release. As far as our users were concerned the only thing 
that changed was that it ran faster; details of hardware architecture were hidden from them. 

With the purchase of an (almost) dedicated DBMS server, we had clearly followed the 
buzzword trend and gone from “distributed" towards a more “client-server" architecture. 


4 A Generational Change: X 

We continued to add 3/60 workstations to the system, and along the way replaced the 3/260 
and 3/60 servers with smaller Solbourne SPARC machines. By early 1991 we had 15 3/60s 
split evenly between Melbourne and Sydney, running SunOS 4.1 and Sunview, and three 
Solbourne SPARC servers. However, there were some huge problems looming. 
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4.1 Problems with the existing system 

The first was that the Sun 3s were (in they eyes of the market) obsolete. Sun had just 
announced that the current OS release would be the last ever for the Sun 3 line. Applications 
were being released for SPARC only with no 68k versions available. The newly-released 
Ingres 6 release was not immediately available for the 68k machines and was always going 
to be a long way behind the main releases. 

The second problem was SunView, which (for all sorts of good reasons) has lost the 
window system wars. The fact that it was host-based meant that the only way to run 
applications of the remote servers was to use rlogin. For text applications, that was fine, 
but for something like Mathematica (which you really want to run on a fast server, for both 
performance and licensing reasons!) getting graphics output onto the local screen was at 
best difficult. Software writers (both commercial and freeware) were likewise abandoning 
SunView. so our choice of products was rapidly diminishing 4 . 

The third was the proliferation of all these workstations, each of which added to the 
load of the system administrator, chewing up time for no useful purpose. 

4.2 Potential solutions 

These problems had a couple of potential solutions. 

4.2.1 Use OpenWindows instead of SunView 

The official Sun-sanctioned replacement to SunView at this point was the hybrid X-NeWS 
product OpenWindows (straight NeWS having already lost the battle for the networked 
window system). This would have fixed all the problems we had with SunView, and 
(because it was network based) allowed us to use SPARC applications programs if we 
needed to. However the OpenWindows release was a performance pig; on the 8 Mb 3/60s 
it was so slow as to be unusable. 

4.2.2 Upgrade the hardware 

Sun were by this time offering pretty good upgrade deals to the Sun 3 customers to upgrade 
to newer SPARC workstations. Running OpenWindows on these upgraded machines would 
have solved all of the problems outlined above, but the cost was quite substantial. (A trade- 
in on a $30,000 machine still leaves a $25,000 machine...) The advantage would be that we 
would have had much more compute power available on the desk. 

4.2.3 Use X-terminals. 

The third alternative was to abandon the workstation model altogether and use X-terminals 
instead. At the very least, re-equipping with X terminals was likely to be much cheaper 
than upgrading all the workstations to Sparcstations. And, as it turned out, we could use 
our existing 3/60 workstations as X-terminals, so this option was even cheaper. We needed 
to determine if the X-terminal and server model was appropriate for our business, or if a 
workstation-based model was more suitable. 


5 The Decision Process. 

Given the problems and the potential solutions, what were the things we did to help us 
choose? 

4 Actually, it would be closer to the (ruth lo say that the choice was remaining static but very low; the choice 
of X-based software was by contrast growing at a huge rate 
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5.1 Know the alternatives! 


At the time we began considering these issues we had no experience using X, and only a 
vague idea what X was and what capabilities it offered. In order to make informed choices 
it was necessary to gain some experience. So in late 1989 we did just that. We got a tape 
of the standard MIT X release, compiled it up, and started using it within the EDP team. It 
turned out that there is a fairly substantial learning curve to negotiate even as a user, with 
things like App Defaults files, resources, keymaps, sessions, clients etc. Then, as a system 
administrator, you need to learn about managing terminals, resource requirements for X 
clients, and so on. By the time we needed to make a decision, we had been running X 
for over six months, and for three months the EDP team had been running an environment 
similar to that we were proposing for our users. 

5.2 Know your Users! 

The second thing is to take a look at your users, and make sure that you understand exactly 
what they want and what they are doing with the current system. We watched our users, 
ran some monitoring scripts and discovered something unusual. The vast majority of our 
users were “read-only.” That is to say, they came in in the morning, logged in, and were 
presented with their saved desktop. From that point on, it was very rare that they started 
another program, and uncommon to use the keyboard at all. What they did do often was 
use the mouse to open and close up to 20 application icons as they needed the information. 
We also determined that most of the applications they were running were text-only and 
basically display-only applications with little or no calculation involved. A look at the 
accounting on the workstations showed that even the 3/60s could easily cope with the CPU 
load, with most of the time spent on the SunView terminal emulator program. 

The main users of the computational-intensive programs were also basically passive 
watchers, with only isolated computing, so the total CPU demand was very bursty. Most 
of the heavy CPU load on the system happened on our main DBMS server. A lot of 
information was pre-calculated and stored in the database, from where it was accessed with 
simple forms-based client programs. 

5.3 Know your Servers! 

We were running two servers in Melbourne. One was handling the DBMS, and occasionally 
running Mathematica or other compute-intensive tasks. It contained all the archive data and 
the source directories. It was also being used by the EDP team for software development. 
This machine was reasonably busy, but short of memory. CPU and disk resources were 
adequate (except that there is never enough disk space!) 

The second server in Melbourne and the corresponding server in Sydney was used as 
the fileserver for the diskless workstations and the PCs, and to do the majority of the quote 
processing. They had a heavy disk I/O load, but the CPU was more-or-less unused, being 
about 80% idle even at peak times. 

We also discovered that the diskless paging traffic from all those client workstations was 
a serious load on the ethernet. This was especially noticeable when the once-per-minute 
index updates came from the real-time source. This would be processed by the main quotes 
program and then forwarded to each of the workstations. The workstations would then 
forward this information to many of the running programs, each of which would be paged 
in to process it. They would then output something to the screens, so all the terminal 
emulators paged in. This caused a storm of paging lasting about 10 seconds once per 
minute. (A similar effect has been noted when running routed or rwhod on diskless 
machines.) 
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6 Implementing it 

All of these factors pointed to a compelling conclusion: Workstations were inappropriate 
for our environment, and X-terminals connected to a quite modest server could easily fulfill 
the needs of our users. Having determined this, there were a number of remaining questions. 

6.1 “Look & Feel” 

The first was one of “look & feel”. There were three major choices here. The first one was 
to use “raw” X, using the Athena widgets for our applications and twm (or similar) as the 
window manager. This was appealing for a number of reasons: 

• The price is right (free). 

• The performance is substantially greater than either Motif or Open Look, at least in 
their current implementations. 

• A much wider choice of more-or-less compatible window managers (c.g. twm, vtwm, 
tvtwm, ctwm, etc) 

• Most “freeware” will use these widget sets. 

The result is however likely to look and feel a bit patchwork and will not have the nice 
integrated feel that is achievable with the other options. In addition, purchased applications 
will in general not use Athena widgets. The performance difference is significant enough 
that the EDP team and one of the more computer-literate dealers still prefer and use twm or 
some variant. 

The other options are Open Look (used by Sun) and Motif (used by just about everyone 
else). Being SPARC based. Open Look would seem the logical choice, at least in terms 
of third-party software availability. However, in our environment, we chose Motif for the 
following reasons: 

• It has a very similar look and feel to Windows on a PC. This gives us continuity 
for users across the entire company, a number of whom switch between PC and 
X-terminal environments. This, in our case, was the clinching reason. 

• You can buy the source license relatively cheaply ($USD2,000). This implies a 
bit of effort in compiling it up and making it work, but it is invaluable in writing 
applications, because the toolkit is full of interesting little gotchas that you would 
never solve without source. The ability to patch and recompile the window manager 
turned out to be vital. 

• The UI builders available for Motif are more varied and much more capable than 
those for Open Look. 

• Most application vendors are now supporting Motif on SPARC, often as the preferred 
interface, so lack of applications is not a problem. 

6.2 The Server 

The one thing X requires is plenty of physical memory. We estimated that about 4-8Mb 
RAM per user would be required, given the large working sets of the X and especially 
Motif-based programs. CPU was not a problem, and ethemet bandwidth was not a problem 
(especially compared to the paging we were currently experiencing). 

We had decided that we had plenty of CPU cycles to spare on our server, but the 16Mb 
memory in the machine would not be enough to support 10 X-terminals. However when 
we came to purchase the memory upgrade, we discovered that K-BUS memory for the 
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Solboume servers was very expensive. So expensive, in fact, that it was much cheaper to 
buy a headless desktop machine (an S4000, about equivalent to an SS2+) with plenty of 
memory (72Mb) and a 400Mb drive rather than upgrading our existing server. So this is 
what we did. We ended up with three servers in Melbourne. The DBMS server remained in 
the same role. The machine that previously was the fileserver for the diskless workstations 
now serves the PCs and handles the real-time processing (and is much less busy than it 
was). The new S4000 acts as the XDM host for the X-terminals. It does nothing but run 
user processes, in particular the window managers, xterm clients and applications. 


6.3 The X-Terminals 

Having decided to go the X-terminal route we had to actually procure some X-terminals. 
There are many to choose from on the market, but to get equivalent screen size and 
resolution to our existing 3/60 workstations, it turned out to be quite expensive, around 
S8.000 - $10,000. Most of the cost for these was in the video display. 

But we already owned perfectly acceptable video displays in the form of the 3/60s. In 
fact, they had all the hardware that an X-terminal needs; all they lacked was the software. 
As it turns out we were not the first people to consider this. Seth Robertson from CMU 
had already looked at this and come up with a solution 5 . He produced a package called 
“XKernel” with was basically a cookbook guide to using diskless 3/50 and 3/60 machines 
as X-terminals. By following this cookbook (outlined below) we were able to reuse all our 
existing workstations in the new environment. 


7 Results 

Having discovered all the above, and bitten the bullet and done it all, how did it work out? 
In short, extremely well. We trialled the new installation for a couple of weeks with one 
or two test users, then changed the remaining users over one weekend. Most of the next 
week was spent tidying up loose ends and retraining users. Since then we have purchased a 
number of other 3/50 and 3/60 machines on the second hand market to use as X-terminals 
and also purchased real Labtam X-terminals. 

Those users who had experience with Windows just started using the new system with 
almost no training. This vindicated our decision to use Motif. 

The 3/60 workstations made excellent X-terminals. They have run very reliably (espe¬ 
cially with the later patchlevels of the R5 release which ironed out a few font server bugs) 
and once installed take no administrative effort. Instead of 30 machines to monitor, I now 
have 5, and it has saved a lot of time. Performance as X-terminals is exceptionally good 
(although I have not measured Xstones or whatever), and feels about as fast as our Labtam 
terminals. Even with only 4Mb of memory 6 the color 3/60s handle the load very well. 

A most unexpected result, but one that our users were most appreciative of, comes as a 
result of the cgfour frame buffers on the Sun machines. These cards have ten bit planes. 
Eight are used as a standard 256-color plane, one is a mono overlay plane, and one selects 
overlay or main plane. Under SunView, this was all used for a single screen. Using the 
standard MIT Xsun you end up with two logical screens on the one display. One ia an 
eight-bit pseudocolor, one a one-bit staliegray. Moving the mouse cursor off the left or right 
edge of one screen does an instant flip to the other screen. This gives our users a whole 
new screenful of (monochrome) data, accessible with the flick of a mouse. It has proved 
to be a popular feature, so much so that there is now competition to get the 3/60 based X 
terminals (rather than the higher-resolution Labtams) because of this capability. 

5 This one discovery, via Usenet news, saved us tens of thousands of dollars and is more than enough justification 
for obtaining network access, despite the cost in phone bills and programmer productivity (or lack of it). 

6 A number of our 8Mb machines have had a SIMM go flaky; in this case we just rip the offending bank out 
and run with 4Mb. 
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Oar decision about the server also proved to be correct. With 72Mb of memory, running 
10 X-terminal sessions each with 10-20 clients, it pages only rarely and uses about 10-20% 
CPU. We estimate we could add another two to four X-terminals immediately, or up to 20 
terminals total with another 32Mb of memory. All this on a server costing under $10,000. 

One area that involved more work than expected was keymaps. The whole handling 
of X keymapping is obscure and badly documented, and Motif also makes a number of 
undocumented assumptions in this area. Getting this right was a major effort, in particular 
getting some sort of consistency across different X-terminal keyboards and the use of 
NumLock keys on PC-101 keyboards (which required a patch to mwm to fix). Each 
different brand of X-terminal has a slightly different default map, and integrating a new 
brand or model can take some effort. 

What we have achieved in the end must certainly fit all the buzzwords of the moment: 
Open Systems, Client-Server, Standards-Based, Distributed. Unfortunately, we didn't have 
a mainframe to Downsize or Rightsize from! In fact, the client-server approach can be seen 
at two levels. The X-terminals are X servers and the user processes on the new machine 
are X clients. Those user processes are also clients of the services provided by the DMBS 
server and the real-time server. 


8 A Quick Look At XKernel 

XKernel is a cookbook designed by Seth Robertson from CMU to use 3/50 workstations as 
X-terminals. His package was put together some time ago, and assumes a single machine 
architecture and an R4 server. Extending this to mutiple architectures and to take advantage 
of the R5 facilities (in particular the font server) was obvious. 

The basic idea is simple: run the machine as a diskless node, with absolutely everything 
that is not needed chopped out. You still need one machine (probably a Sun but not 
necessarily) somewhere on the local net with about 8Mb of disk space to act as the boot 
server. I will assume some reasonable familiarity with configuring and booting diskless 
Sun machines. 

The first step is to make a diskless kernel. Start with the DL50 or DL60 config file, 
and chop out just about everything and set maxusers=l. This produces the minimum sized 
NFS-booted kernel. 

The second step is to get the MIT X source and compile up a statically linked version of 
Xsun. If you are using R5 (and you really should) you can edit one line in the Sun-specific 
init files to remove support for reading fonts from the filesystem (this saves about 80k). 
You can then use the font server running on the boot host for fonts. If you have 3/50s or 
mono 3/60s you also want a statically-linked XsunMono (which saves about 100k). Don’t 
include the PEX extension unless you really need it - it adds about 500K. 

Make a single swap directory for all the Xkernel machines. Use mkf ile -n 4m to 
make a 4mb swap file without allocating any blocks. It can be mode 0444 because no-one 
writes to it. 

For each kernel architecture, make a root directory 7 . In our system, this involves 
three directories called XKernel, XKernel-50 and XKernel-80 for 3/60s, 3/50s and 
3/80s respectively. In each root directory make dev, etc, tmp directories. Copy in a 
MAKEDEV script and make the standard set of devices. Copy the relevant kernel and a copy 
of the appropriate Xsun binary. Create an sbin directory and copy the contents of /sbin 
into it. (You can save a few Mb of disk by hard-linking the various copies of binaries in the 
various root directories.) 

The next step consists of compiling the supplied 20-line replacement init program and 
installing it in sbin. This init program does nothing but exec the /etc/rc.Xboot 
script. This script can be as simple or as complicated as you like. In our environment it 

7 This is necessary because each architecture will have a different kernel, and there is no way for bootparamd 
to specify which kernel to boot without editing the EEPROM. 
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looks at the host name to decide which Xsun version and with which options to run (in 
particular, which server host to request XDM service from), then exec’s Xsun. 

It should be noted that the Xkernel machines require no /usr partition; the only system 
binaries they use are in sbin. However, they are still running SunOs so you will still need 
a SunOs license (which is not a problem as an RTU comes with the machine). Once booted, 
the only process running on the machine is the Xsun server. 

Add enough of art etc/hosts file to find all the hosts named in the boot script and 
a minimal etc/f stab that mounts / and swap read-only. Then, on the boot host, make 
sure the ethers, hosts and bootparamd files/YP maps are up to date, make the symlinks in 
the / t f tpboot directory, and away you go. 

In our environment, the total disk space to support 10 X-terminals of four different 
architectures is 8Mb. 

9 Conclusions 

I should start by emphasising that our user base may be atypical. Perhaps not many 
enviroments have such a majority of “read-only” users. But my suspicion is that this is 
much more common than vendors of workstations or servers would like us to think. 

Open Systems really has made an impact in the area of selecting hardware. With product 
lifetimes of three years or less, obsolescence simply cannot be avoided, and changing the 
underlying hardware architecture is a fact of life. Running heterogeneous environments is 
certainly possible and (at least if they are all flavours of Unix) only a minor inconvenience, 
but it does require extra disciplines from the EDP team. 

However, if your applications are build on some purchased foundation technology (in 
our case, the Ingres RDBMS), then your hardware choices will be limited by whatever your 
software runs on. The lesson here is to be very careful about choosing this software because 
you will be stuck with it more-or-less forever, but your initial hardware purchase has a life 
of a few years at most. 

The Unix desktop market has split into two “us and them” camps, with the Workstation 
manufacturers on one hand and an alliance of X-Terminal manufacturers and “supermini” 
server vendors on the other. Each will claim that their architecture is the wave of desktop 
computing in the 90s. The truth of the matter is that neither is correct 100% of the time. In 
some environments (such as ours) desktop CPU performance doesn’t matter, but low-cost 
access to large bitmapped screens and GUI ease of use is what we are after. X Terminals 
are much more cost-effective for us. Conversely, an environment that was doing serious 
bit-shuffling (e.g. modeling, 3d graphics, perhaps even desktop publishing) would be better 
served with at least some workstations. My suspicion is that (to Sun’s disappointment) these 
are by far the minority of GUI desktop users. To take an example that is perhaps relevant 
to this audience, in a software development environment I would be much happier with 
a single large central machine, where all the source is stored, and a bunch of X-terminals 
rather than a collection of workstations. If nothing else, compiles will be quicker if they 
are to local disks! 

Even amongst X-Terminal vendors, the emphasis is often on performance rather that 
resolution. We often found the low-end slow machines had at best VGA resolution, and 
to get megapixel-level resolution you had to buy the high-end zillion Xstone model. And 
any vendor who comes out with an X-terminal that has the two-screen mode that the Sun 
X-terminals have will get money from us and I suspect plenty of other people. 

Reusing the obsolete 3/60 technology has worked extremely well, and will continue to 
be a valuable strategy. At this point in time, the early SSI machines are approaching the 
end of useful life, and the same strategy can be applied with these machines. Indeed, I 
know of one vendor who will shortly start selling SSI-class clone machines explicitly as 
X-terminals using just this strategy. 
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Finally, and it might seem like a truism, don’t believe everything you vendors tell you, 
and make sure you understand exactly what your users do and what they want. 
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Report on Usenix 
San Diego 25th-29th January 1993 

Greg Rose 

Australian Computing and Communications Institute 
ggr@acci.com.au 


I arrived in San Diego from New York on Saturday. The main reasons for me to arrive 
early were firstly, because the Usenix Board of Directors meeting was scheduled for Sunday, 
and secondly, given the choice between New York or San Diego in winter, well... 


Sunday... Board Meeting 

Attendance at Usenix’s “big" conference is declining, and this is seen to be a worrying thing, 
but it is most likely to be because the associated smaller conferences are doing extremely 
well. For example, the Large Installation System Administration conference, and the C++ 
conference, are both about half the size of the main conference. The program for this 
particular conference is seen to be extremely strong, especially with a number of great talks 
from Australia. 

Elizabeth Zwicky reported about SAGE (the System Administrators’ Guild), Local 
Technical Groups (Regional groups), Special Technical Groups (SIGs), and International 
Affiliate Groups (eg. Sage-AU). More about Sage later; for now suffice it to say that Usenix 
is happy that Sage-AU is up and running, and wants to cooperate. 

Usenix had planned an Application Development Symposium, but that was cancellled. 
It was supposed to be a joint venture with UniForum Canada, and the logistics got too hard, 
but there was a reasonable amount of interest. “Symposium" was probably too high-falootin 
a word for this thing. This is something I think AUUG could do well. 

Finally, after a number of years of being shunned, a change of management at UniForum 
is now amenable to getting friendly with Usenix again. Initially, this cooperation will 
probably take the forms of jointly funding the POSIX standards watchdog function, and 
hosting a major Usenix event (possibly the next LISA) at the same time and place as the 
UniForum in San Francisco, in early 1994. 

Personally, I think these previous points show that AUUG was right to try and avoid 
splitting into technical versus commercial subgroups. UniForum has bet their farm on 
Unix, and really are not in a position to move towards other “open" systems. Usenix on the 
other hand have diversified quite successfully, but rely probably too much on the moral and 
technical high ground. Now they seem to need each other again. (End personal opinion.) 

Mick Farmer reported on the new structure of EurOpen. Like most things in Europe, this 
group (formerly the European Unix Users Group) is undergoing rapid and disruptive change. 
EurOpen was structured as an umbrella organisation, but also undertook to organise their 
own conferences to some extent. The Secretariat was moved from Owles Hall in England 
to Brussels, Belgium, and a buch of employees were terminated at the same time. A couple 





of member countries refused to pay their dues, and EurOpen was looking at bankruptcy 
this year. So, a quick restructuring resulted in “EurOpen Lite” which seems to merely 
coordinate and distribute information between the member country’s groups (and no longer 
competes with them). It is too soon to say whether this will succeed or not, but it is a valiant 
effort to save the otherwise doomed organisation. AUTJG seems better all the time... 

Usenix’s Board sere all surprised to hear that 32% of their members were from overseas 
(which doesn’t include Canada). Peter Collinson (U.K.) and I, who regularly attend these 
meetings, had a bit of a giggle about this, as we have certainly known it for some time. 

When the board meeting moved into closed session (i.e. they threw out the hangers-on 
like me) I went around to the registration area and picked up my books etc. It was within the 
first few minutes that I realised what the really hot topic of this Usenix conference would 
be. 


Background: Unix System Laboratories is sueing the University of California at Berke¬ 
ley, and Berkeley Software Design, Inc., over some sort of alledged infringement of licenses 
or copyrights or look and feel, or something equally intangible. In the latest round of this 
war, USL required a list of all people who have ever worked for a Unix Source code 
licensee, or who have ever seen source code of Unix or of the Berkeley NET-2 (re-written) 
release, or who have read anything about internal design or data structures of Unix (ever 
read the Bach Book, or the Lions Commentary?). The reason for this request is that USL 
asserts that any code such a person writes for an operating system, or in which the same 
algorithms are used (e.g. linear search, used extensively in Unix) is really the property of 
USL. Their deposition used the words “mentally contaminated” to describe such people. 
The court is currently deciding the status of this suit. (End of Background.) 

Rick Adams, who works for BSDI and UUNET, has made a bunch of badges which say 
“MENTALLY CONTAMINATED” in large red letters, and I happnend to have a pocket 
full that I was intending to return to Australia. Virtually everyone who saw the badge said 
that they were contaminated, and so wanted such a badge. I ran out within minutes. (Don’t 
worry, I got more.) Peter Salus was wearing the UKUUG windcheater with the famous 
“/* you are not expected to understand this */” comment, and pointed out that anbody who 
looked at him was also forever contaminated. (In fact, if you just read that, you are now 
contaminated too... fortunately most of this audience is in Australia and we appear to be 
sensible enough that we can ignore this stupidity. The judge here (theU.S.) is expected to 
rule on this point after about two weeks of deliberations, and no one can guess which way 
it will go.) 

The “launch” of the week was held at 18:00 and featured non-alcoholic drinks free, a 
cash bar, and some quite good (and sometimes very spicy) mexican munchies. 

The conference T-shirt features a standard sort of San Diego advertising picture, with 
a caricature of Rob Kolstad (Conference Convener) surfing on an oversized keyboard. 
Tomorrow night there is a group getting together to sew pink tutus around him (sorry, an 
“in” joke - it was proposed on the network that there should be a competition about Rob 
and tutus). 


Monday... Tutorial day 1 

Not much to say about this day. There are about ten full day tutorials each day, and I attended 
the one about OSF DCE. The standard of the tutorials is pretty high, even if the information 
is something you would rather not hear. DCE, despite its total lack of real availability at 
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this point in time, is already an important standard. Fortunately I was already familiar with 
all seventeen of the possible synchronisation primitives which are all supported by DCE, 
so 1 floated around meeting people for the morning and only sat in on the DCE tutorial for 
the last half. Dinner was a rather nice mexican meal from somewhere north of UCSD. 


Tuesday... Tutorial day 2 

Relatively little to report on this day, as I decided that I needed to fill in the gaps in 
my knowledge of Kerberos, and the tutorial by Dan Geer and Jon Rochlis was excellent. 
Tuesday evening was the board meeting for USENIX/SAGE which I attended representing 
Sage/AU. 

By the end of this day, my conference badge was getting rather heavy. As well as the two 
ribbons I deserved (Invited Speaker and Conference Pilot) I had added a “Newcomer", as it 
was such a pretty yellow. I had a dinosaur sticker, the “MENTALLY CONTAMINATED" 
warning mentioned above, and a similar pin in the form of a New Hampshire number 
plate, saying “NET2", and with an insertion mark and AT&T inserted in the motto, vis 
“Live (AT&T) Free or Die". At the end of each ribbon was a button. One was the “Robbie 
Kolstad fan club", another a Henry Spencer badge quoting Marshall Rose “OSI Committees 
lack Adult Supervision", and yet another lawsuit badge, “NET2" circled by the words “Want 
To Be Sued? Ask AT&T How!". This badge was getting pretty heavy. The solution to the 
problem appears on the next day... 


Wednesday... Conference Day 1 

The keynote talk was “Pen Based Computing and its Impact", by Robert Carr of GO 
Corporation, writers of the PenPoint Operating System. This talk was quite interesting, 
until the last few minutes, but I'm not sure that it deserved the keynote position. I can 
believe that the interace to the computer through a pen might be more productive for many 
people, particularly people moving around a lot. The last five minutes of the talk, however, 
were mind blowing. All the features are usable in Japanese, including the handwriting 
recognition. Watching a video of japanese characters being written and then redrawn 
accurately after being recognised was impressive. 

There were two minor problems with the talk. The speaker claimed that PenPoint was the 
first operating system to support Unicode, at which Dave Presotto and Phil Winterbottom, 
the implemented of Unicode in Plan-9, sort of choked, and they called the Hobbit chip 
from AT&T a “brand new high performance RISC microprocessor", while it is in reality 
the same old CRISP chip revisited. 

Immediately afterward, Rob Pike gave a talk about use of Unicode in Plan-9. I can’t 
reproduce the title here, as it included japanese and Hebrew versions of “Hello world". 

Many of the papers on this day were not of great interest to me, and because of the 
weight of my badge, I decided to start a new business (profit free) producing funny name 
badges. Because Rob Kolstad is the current scapegoat, name badges like Greg Kolstad, 
Ken and Dennis Kolstad, Kirk Kolstad, and even Rob (Pike) Kolstad appeared. 

Dinner was an excellent psuedo-italian meal while Addison Wesley negotiated a contract 
with me for my new book (that’s a plug). 

This evening, there was an Open Board Meeting for Sage, in which it was announced 
that there was an intent to affiliate between Sage/US and Sage/AU. The committee for 
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Sage/US is Steve Simmons (President), Pat Parsegian (Secretary), Peg Schafer (Treasurer), 
Carol Kubicki, Pat Wilson, and Paul Moriarty. There was an observation that there was 
an enormous momentum behind the formation of Sage, that the first year was extremely 
successful, and that there was a great level of excitement about the future of Sage. 

Sage has a number of working groups and discussion groups, which are displaying 
varying degrees of success in achieving their goals, or even specifying their goals. There 
is now a sub-committee which will review these working groups. 

What do you get for your membership in Sage? An investment in the future, and in the 
working groups. A chance to be heard as a coherent group, without being drowned out by 
the other voices. Gradually, Sage will take over the Lisa conferences. 


Thursday... Conference Day 2 

There were a number of interesting papers given, including a talk from Plan 9 about the 
I/O system. Some people from Bell Labs had been, shall we say aggressive, to some of 
the earlier presenters, so they came in for some bashing of their own from Peter Honeyman 
and me. 

Dan Klein gave an excellent invited talk about languages for specifying specialised 
things, and the major example was the specification language for Blazons (medaeval shield 
designs), with parallels to PostScript. 

Immediately after lunch there was a highly acclaimed talk about “A History of Unix”, 
and despite lots of people in the audience who had been closer to the events than the humble 
author, there were few disagreements over facts. 

The conference reception was different to previous Usenix events, in fact it was much 
more AUUG-like, with tables and entertainment. Unfortunately, the impromptu comedy 
foursome did a pretty poor job of making jokes about relevent issues, and even got Ken 
Thompson’s name wrong (Ritchie Thomas!) in one of the jokes. The jokes that weren’t 
Usenix specific were pretty smutty and sexist, and didn’t go over well. I think it will be the 
last time Usenix tries that idea. 

After the reception, and with the pressure of my talk over, I proceeded to get (more) 
drunk, culminating in the Single Malt Scotch Working Group meeting, which went till well 
after midnight. That’s why the typing is not terribly coherent. 

This year, Usenix has introduced an annual ‘’Keepers of the Flame” award, and the first 
one was presented to the Computer Science Research Group at the University of California 
at Berkeley. The seven members of the CSRG get rather nice glass sculptures, and corporate 
contributors get plaques. Then there was a huge list of major contributors who will get 
calligraphed certificates of appreciation, and Robert Elz was deservedly on this list. Then 
there was a list of about 160 other significant contributors, with a number of Australians on 
it. This is a nice idea. 


Friday... Conference Day 3 

Andrew McRae opened the day, and his talk was well received by those who attended. I 
was in the parallel invited talks track, where some of the best papers from the filesystems 
workshop were represented. Peter Honeyman from the University of Michigan was the 
host and presented one of the papers, about Alex, the NFS FTP filesystem. 
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On one of the previous days, Margo Selzer presented a joint paper about the results of 
the final implementation of the Log Structured File System. This has been one of the great 
hopes on the horizon for improving the performance of file systems, but this paper basically 
canned it, pointing out that the overheads of garbage collection eventually soaked up most 
of the benefits for common scenarios. 

As an obvious corollary to the promising initial results, this same conference had a 
number of papers about novel applications of the log-structured filesystems, which were no 
longer such wonderful things to do... 

At lunch time I got back to work and attended a meeting of Usenix’s tutorial program 
committee, to get ideas for tutorials for the coming AUUG conference. (I am tut coordinator 
for AUUG’93.) I have some good ideas, if people want to present a tutorial but don’t know 
what to talk about. 

As I mentioned earlier, there had been some amount of on-going Rob Kolstad bashing, 
particularly with reference to pink tutus. At the beginning of the after lunch Invited Talks 
session, which was a representation of the best papers from the last Lisa conference, this 
was finally laid to rest (perhaps). Steve Simmons began by introducing Rob (whose badge 
by now said simply “Rob *”) and explaining the origin of the whole tutu business. He then 
said that Rob would be presented with a pink tutu which he could ceremoniously burn, and 
end the matter. What he didn’t mention was that the tutu in question was being worn by Ed 
Gould (190 cm tall, 140 kg wide, and hairier than me). This raised quite a laugh. 

At the close of the conference two equal awards were presented, for the best presen¬ 
tation. One award went to Margo Seltzer, Keith Bostic, Kirk McKusick, and Carl Staelin 
for “An Implementation of a Log Structured File System for UNIX’’, despite the depress- 
ingly negative result. The second went to Stephen Uhlcr for “PhoneStation, Moving the 
Telephone onto the Virtual Desktop”. 

After the conference was over I went to dinner with some excessively rowdy people 
from Bell Labs, and then to a party. This was a hard week for me. No, really, I didn’t leave 
the hotel grounds except for two dinners. I only got to the hot tub once... 


Overview of Content 

Given three parallel tracks, and the fact that there was never a clear distinction in the type of 
the content, obviously it was impossible for me to see all the papers. I’ll mention here those 
that I thought were worthy of specific comment. I will suggest to the newsletter editor that 
a full table of contents be printed, and you can purchase copies of the proceedings through 
AUUG. 

I’ll repeat, for the record, the observation that it was an extremely good technical 
program. 

“Pen based Computing and its Impact”, Robert Carr. 

See discussion above. 

“Hello World” (even Usenix proceedings shortened the title), Rob Pike. 

Describes the use of a variant of Unicode as the base character set for the operating system, 
and some of the issues that arose during the implementation. 
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“DUEL - a Very High Level Debugging Language”, Michael Golan and David R 
Hansen. 

The authors present a small language which is oriented towards evaluating expressions 
about high level data structures. It is interfaced to a common debugger (dbx?) and finds its 
chief use for writing things like “show me the elements that are out of order in this array”. 

I find the language interesting and useful, but the application inappropriate. 

“PhoneStation, Moving the Telephone onto the Virtual Desktop”, Stephen Uhler. 

You had to be there, but reading the paper is also very rewarding, particularly if you have 
Sun SparcStations. 

“Jgraph - a Filter for Plotting Graphs in Postscript”, Janies Plank. 

I only caught the last part of the talk, but this looked like useful stuff well implemented. 

“VVafe - An X Tooklit Based Frontend for Application Programs in Various Program¬ 
ming Languages”, Gustaf Neumann and Stefan Nusser. 

The X application builder wars are being fought between Tk and Wafe. Stuff about Tk 
appeared a couple of Usenixes ago, and if you do any X application programming you need 
to know about both of these, although you will probably end up settling on one. 

“The Design and Implementation of the Inversion File System”, Michael Olsen. 

Inversion is a file system, accessible through NFS, that is implemented within the PostGres 
database (hence the name), and has all sorts of interesting properties regarding uncor- 
ruptability, consistency and recovery, not to mention some interesting semantic properties 
(Query-language constructs in pathnames!). 

“File Systems in User Space”, Paul Eggart and D Stott Parker. 

How (not) to hack useful but irregular and unpredictable semantics into your tile system, 
by totally subverting shared libraries. Read it, the idea is very interesting, but the design... 

“The Organisation of Networks in Plan 9”, Dave Presotto and Phil Winterbottom. 
Worth reading. Watch for things by Phil Winterbottom. 

“An Implementation of a Log Structured File System for UNIX”, Margo Seltzer, Keith 
Bostic, Kirk McKusick, and Carl Staelin. 

As usual, a right scholarly piece of work that analyses the performance of the much touted 
Log Structured File System in the real world. Unfortunately, it seems perhaps to have been 
a dead end. There was a paper a year or so ago that used a different method to obtain faster 
write performance, that perhaps now need more examination. 

“The Nachos Instructional Operating System”, Wayne A Christopher, Steven J Proc¬ 
ter and Thomas E Anderson. 

I didn’t see the presentation, but the paper won the best paper award. 

There were lots of other papers that I hope I am not slighting by leaving them out of 
this list. There seemed to be something for everyone at this Usenix. 
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A Case Study in Tracing Real-Time System Events 

Ken J. McDonell 
Systems Technology Laboratory 
Pyramid Technology Corporation 
kenj@pyramid.com 

Abstract 

In the context of quantifying program behaviour for studies of new architectures, we became 
interested in the dynamics of System V Release 4 (System V.4) shared memory usage. In 
particular we needed to establish the distribution of inter-arrival times between shared 
memory accesses, and the relative frequencies of access modes (read or write), access lengths 
(byte or word, scalar or array) and access types (CPU or I/O). Specifically, this 
characterization was required for DBMS applications, meaning the event rates could be quite 
high. 

The implemented solution involved some specific kernel techniques in the areas of process, 
memory and address space management, coupled with some more general services to support 
kernel-level event tracing. To achieve efficient and correct operation, the software 
implementation had to accommodate the realities of hardware cache behaviour, a light¬ 
weight solution for mutual exclusion in critical code sections, re-entrancy control and bulk 
data export from the kernel. 

In addition to the kernel implementation some user-level code was required to post-process 
the event traces. 

1. Background 

There are four commonly used techniques for tracing events. 

1. Instrument the user’s code - not useful for shared memory accesses because (a) 
instrumentation would be omnipresent, and (b) we did not have source code for the user 
code (it tended to be things like Oracle and Informix!). 

2. Instrument the system call interface, or use tools akin to trace( 1) or truss( 1) - not 
helpful for shared memory accesses in user code! 

3. Use a hardware monitor - this would offer minimal performance intrusion, but is not 
realistic for shared memory tracing because, 

a. it may be difficult to capture the context of a shared memory access, e.g. initiating 
process ID, CPU versus I/O access, physical addresses versus shared memory 
virtual addresses 

b. need to monitor bus-level (hard) and cache-level (almost impossible) events 

c. bus-level hardware monitors are expensive and difficult to set up 

d. limited memory in the hardware monitors for trace storage (typically less than 10 
Mbytes, which is one or two orders of magnitude too small). 

4. Software tracing by instrumenting the operating system kernel - since this is the last 
option, and the others have been eliminated, this is a good candidate! 

For the most part this paper deals with Unix (System V Release 4) kernel-level details, 
although no prior knowledge of any Unix kernel is assumed. 
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2. The Basic Approach 

At the “white board and hand-waving” level, the strategy is as follows; 

1. When a user-mode page fault occurs, check to see if it is a page in a shared memory 
segment, and if it is 

• log the details of the access, and 

• allow the process to single-step in user mode 

2. When a read( 2) or write( 2) system call 1 occurs, check to see if the I/O buffer is in a 
shared memory segment, and if it is 

• log the details of the access, and 

• allow the process to single-step in user mode 

3. When a process enters the kernel after a breakpoint trap, if the breakpoint results from 
the single-step tracing of a shared memory access, i.e. 1. or 2. above, then 

• undo all page table changes associated with the previous access (mark all effected 
pages as invalid, but do not free the underlying physical memory page), 

• remove the single-step breakpoint, and 

• continue execution in user-mode 

Note that we do not attempt to trap kernel-mode page faults for shared memory pages, nor 
any page faults caused by DMA or other I/O to/from shared memory. 

3. Some Matters of Detail 

3.1 Miss One, Miss ’Em All 

If the page table entries for a process ever end up with valid entries for shared memory pages 
(except in the fault-single-step-trap window outlined above), we risk losing all future CPU 
initiated shared memory activity for those pages. The problem is even murkier when one 
considers that if the page table entries get into this state, they are likely to stay that way for 
all page table entries shared or copied across &fork(2). 

For this reason, we have to be careful to ensure that all possible scenarios for accesses to 
shared memory pages are identified and trapped. 

3.2 How Do You Identify a Shared Memory Access? 

With great difficulty! 

In System V.4 this involves back-tracking from the target virtual address to find the enclosing 
address segment, then eliminating address segments that have an associated vnode (text 
segments and data segments for user code or shared libraries, and mmap{ 2)ed files) or address 


1. Actually we need to worry about other system calls, e.g. ioctl( 2), and other parameters (that might be in 
shared memory) being passed across the system call interface; but this is a mere detail of enumeration, 
not added functional complexity, and even then we can make some assumptions about the way the 
traced applications behave, and ignore some of the unlikely scenarios. 
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segments that do not have an “anon” map (can have no interesting pages attached) or are not 
of a special sharing flavour. 

3.3 Per Process State 

We need to pass some state between the page fault interception code and the breakpoint trap 
code; specifically we need to be able to indicate that the breakpoint trap was caused by the 
shared memory tracing facility and not some dopey debugger or random idiot user code, and 
we need to be able to save the first virtual address and length of the shared memory access so 
we can lobotomize the correct page table entries (see Section 3.4 below). 

Potentially, this information must be saved for each process - we choose the much abused 
“u” area because (a) it is always addressable in a process context with little overhead, and 
(b) it carries so much baggage that it was not hard to find a couple of moribund fields that 
could be silently over-loaded without breaking anything else or recompiling the universe. 

3.4 Undoing a Page Fault 

Because of the way address spaces and their associated page tables are built in System V.4, 
the page-fault mechanism used to detect a shared memory access by a CPU is really a miss in 
the TLB (Translation Look-aside Buffer). 

After the breakpoint trap, we need to ensure that subsequent accesses to the same page(s) will 
also cause a TLB miss. This involves locating the page table entry (or entries), clearing the 
valid bit and removing the virtual to physical mapping from the TLB. 

3.5 Log What? 

For each shared memory access we store a trace record containing; 

• an event type - one is assigned for shared memory tracing, but the mechanism is also 
used to trace other system events, producing a single trace (possibly for a variety of event 
types) 

• timestamp (using a microsecond clock that is accessible from kernel mode) 

• trace parameter 1 — process ID for shared memory tracing 

• trace parameter 2 - first virtual address for shared memory tracing 

• trace parameter 3 - for shared memory tracing we pack both the access mode (CPU fetch, 
CPU store, I/O read or I/O write) and the length of the access (bytes) in here 

3.6 Log Where? 

We maintain a physically contiguous, but logically circular log buffer of trace records for 
each CPU. 

By allocating a private buffer for each CPU and making the buffers aligned on hardware 
cache boundaries we avoid both hardware and software MP (multiprocessor) interference as 
the buffers are filled. 
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3.7 How Do You Extract the Trace Records? 

We require a low overhead interface for extracting potentially a large number of trace records 
from the log buffers into the address space of a user process 2 . 

Pyramid’s System V.4 version (DC/OSx) includes a special statis( 2) system call that may be 
used to extract arbitrarily sized statistics structures from the kernel (this is the “object- 
oriented” approach to hiding a /dev/kmem reader behind a procedural interface that returns 
polymorphic object instances); e.g. the DC/OSx versions of sar( 1) and vrnstat/ 1) use statist2) 
rather than reading /dev/kmem or /proc/mumbleinfo. 

A simple extension to statis{2 ) supported the definition of an input parameter to define the 
user process’ buffer to receive trace records and the necessary logic to accumulate and return 
the trace records from the generic kernel event tracing subsystem. 

To meet the requirements for low overhead, the interface allows for the export of a finite set 
of trace records in a single call to statis/ 2); if insufficient records are available, the caller is 
blocked. To minimize sleep§-wakeupQ thrashing and to allow concurrent processing of a 
partial set of trace records, the caller may set a low-water threshold and statist 2) will return 
as soon as at least this many recods have been accumulated and no further records are 
presently available. 

We avoided many of the nasty synchronization problems in the statis{ 2) implementation by 
making some simplifying assumptions that do not impact the functionality of the service. 
Each trace record is returned to at most one user process - sane applications of this service 
would only entertain a single process to extract trace records. This means that statis( 2) calls 
to extract trace records are treated as a single-threaded critical section, with subsequent calls 
not blocking but failing (ermo == EBUSY) if another process is currently in, or blocked in, 
the routine that accumulates and exports the trace records. 

Since we have N instances of the log buffer for an N CPU system, the extraction routines 
remember where they were up to last time and empty the buffers in a round-robin fashion. 
This means the exported trace records are not in strict chronological order 3 , but we do have 
the opportunity to insert meta-records into the exported trace to identify on which CPU the 
following set of trace records were collected (this data can prove invaluable in subsequent 
analysis). 

Since the trace records could potentially be added to the log buffers faster than we can extract 
them (although this has never been observed) and the producer side never blocks, we have 
the potential for losing trace records. Each trace buffer is preceded by a control block that 
allows us to detect “wrap around” of the circular buffer, and when trace records are “lost” a 
meta-record is inserted into the exported trace to indicate how many trace records were lost. 


2. For lower level tracing (e.g. instruction traces) it is often necessary to force the log buffers to be 
dumped to a raw disk using I/O initiated from kernel mode (to reduce latencies and avoid data 
copying). However, the volume of trace data for kernel event tracing in general (and shared memory 
tracing in particular) is much lower, and so we can tolerate the less efficient, but more convenient, 
scheme of exporting the trace records to a user-mode process, which may well chose to save them in a 
file and therefore force the data back into the kernel via a write/ 2). 

3. But a simple push-down automaton can post-process the records to sort them very quickly! 
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3.8 Turning Tracing On and Off 

Events that may be traced are grouped hierarchically, and event tracing may be selectively 
enabled or disabled for individual event types or groups of event types. This is done by a 
/dev/kmem reader/writer that sets and clears bits in an event trace mask. 

The same control program is able to use a special statist 2) control function to force the log 
buffers to be cleared, and the regular statist!) functionality is used to extract the trace records 
and display them on standard output or save them in a binary data file. 

3.9 Concurrent Access Games 

We have several problems here. 

Code Re-entrancy and Pre-emption 

For statis( 2), this is banned by decree (see Section 3.7). 

When trace records are being added, the allocation of slots in the log record buffer 
is controlled by a conventional spl() mutual exclusion, i.e. disable interrupts for 3 
instructions. Because there is one circular log buffer and control block per CPU, 
there is no MP synchronization required. 

Symmetric Multiprocessor (SMP) Considerations 

Aside from the mutual exclusion in statist 2) (which requires a private counter to be 
protected by an MP spinlock 4 ), the implementation of the event tracing mechanism 
has avoided most of the ugly issues of SMP concurrency. 

Producer-Consumer Synchronization 

When a process tries to extract trace records across the statist 2) interface, and 
insufficient new records are available, the requester (consumer) process sets a 
global flag and blocks via sleept). 

Each time a new trace record is added to a log buffer (the producer operation) we 
check the global flag, and if one or more process is blocked, reschedule them via 
wakeupt). 

The frequency with which trace records are generated is sufficiently high that we 
choose to ignore the potential race-condition and consequent stand-off; if I miss a 
wakeupt), it will happen next time a trace record is inserted into the log buffer. 

4. Performance 

The generic event tracing mechanism is very efficient. 

But in the particular case of tracing shared memory events, TLB-miss (page) faulting is 
expensive. Obviously the effect depends upon the steady-state rate of shared memory 
accesses, but we have observed degradation from 10% (what, me worry?) to a times 80 slow 
down (ouch!). 


4. Spinlocks are busy-waiting mutual exclusion primitives that (when well-designed) are friendly to the 
CPU/memory bus(es) and the data caches in MP systems; spinlocks are used for short-term, non- 
blocking mutual exclusion, e.g. to update a global counter or pointer. 
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5. Sample Results 

Once extracted from the kernel, the trace records are usually sorted and stored in a 
compressed binary format. Subsequent processing by statistical tools can yield summaries, 
of which the following is typical. 

Note that the average sampling rate was approximately 3700 shared memory events per 
second. 

Start: Tue Dec 1 09:57:08 1992 

End: Tue Dec 1 10:06:08 1992 

Elapsed: 9:00 
Records: 2000000 


Pid 

Op 

Misc 

Count 

Serial Byte 

Count #Seq Avlen 

Serial Word 

Count #Seq Avlen 

Total 

Count 

10357 

fetch 

44 

0 

0 

0 

525 

7 

75 

569 


store 

15 

0 

0 

0 

1054 

4 

263 

1069 


read 

0 

0 

0 

0 

0 

0 

0 

0 


write 

0 

0 

0 

0 

18432 

3 

6144 

18432 

13199 

fetch 

295320 

10151 

2958 

3 

34697 

11359 

3 

340168 


store 

151037 

0 

0 

0 

14248 

1641 

8 

165285 


read 

0 

0 

0 

0 

221184 

108 

2048 

221184 


write 

0 

0 

0 

0 

151552 

74 

2048 

151552 

13200 

fetch 

285555 

9876 

2877 

3 

33073 

10943 

3 

328504 


store 

147150 

0 

0 

0 

13107 

1518 

8 

160257 


read 

0 

0 

0 

0 

202752 

99 

2048 

202752 


write 

0 

0 

0 

0 

147456 

72 

2048 

147456 

13203 

fetch 

289380 

10078 

2935 

3 

35073 

11316 

3 

334531 


store 

148926 

0 

0 

0 

15259 

1730 

8 

164185 


read 

0 

0 

0 

0 

233472 

114 

2048 

233472 


write 

0 

0 

0 

0 

149504 

73 

2048 

149504 

13204 

fetch 

295824 

10203 

2971 

3 

34109 

11297 

3 

340136 


store 

151113 

0 

0 

0 

13456 

1565 

8 

164569 


read 

0 

0 

0 

0 

225280 

110 

2048 

225280 


write 

0 

0 

0 

0 

151552 

74 

2048 

151552 

Total 

fetch 

1166123 

40308 

11741 

3 

137477 

44922 

3 

1343908 


store 

598241 

0 

0 

0 

57124 

6458 

8 

655365 


read 

0 

0 

0 

0 

882688 

431 

2048 

882688 


write 

0 

0 

0 

0 

618496 

296 

2089 

618496 
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Abstract. 

In recent, years computer viruses have received much attention although their theoret¬ 
ical aspects have been neglected. One such aspect is the possibility of viruses as life, with 
the attendant characteristics of terrestrial life forms including evolution. Where this par¬ 
ticular idea has been previously considered, it has been judged highly improbable. The 
author disagrees and argues that viral evolution may not only be possible but inevitable. 

After an introduction to the biology of computer viruses evidence of viral evolution will 
be presented as well as examples of how current and future trends in computing could 
lead to the emergence of sophisticated and novel entities. Finally the author speculates 
on the consequences of this. 

1. Introduction 

“If during the last thousand seconds you have received any High-Beyond 
protocol packets from L Arbitration Arts’ discard them at once. If they have 
been processed, then the processing site and all locally netted sites must be 
physically destroyed at once. We realize that this means the destruction of 
solar systems, but consider the alternative.” 

Vernor Vinge, A Fire Upon the Deep 

Less than a decade ago, there was no term to refer to what we now call c computer 
viruses”. Since that time we have seen an escalation that has matched, and in some 
ways, surpassed the intrusion of computers into our lives. 1 The extent to which viruses 
concern computer science in the present day can be measured by the economic impact 
of major viral outbreaks, some of which have been gauged at costing millions of dollars. 
Defence against viruses forms a major part of any concern regarding sensitive or critical 
data. Furthermore the design and detection of viruses has, apparently, attracted the 
efforts some of the more talented minds in information technology. 

However, it is a more speculative but no less important prospect that occupies this 
paper — the idea of viruses as evolving lifeforms. The author proposes that viral biology 
and present trends may be leading towards the spontaneous emergence of wholly new 
and highly sophisticated forms of virus that may confront us with our first examples of 


1 0ne recent work on viruses traces the developments in the field on a month by month basis [1]. 





alien life. Before considering this, mindful of the background of potential readers, we 
shall first examine basic virus function. 2 

2. Virus biology 

2.1. What is a computer virus? 

Basically a computer virus could be defined as any segment of a larger piece of computer 
code that replicates itself. There are a number of entities that fit this description or are 
commonly associated with viruses. The best known is the worm , an autonomous and 
mobile replicating process. Others include Trojan horses (that masquerade as legitimate 
programs whilst carrying out unintended operations), logic bombs (which lie within 
legitimate programs and when triggered carry out malicious functions), chain letters 
(self-propagating electronic mail) and rabbits (programs which endlessly replicate until 
system failure). Viruses differ from these in several significant ways. First, replication 
is purposeful and not simply automatic. Second, the actions of the virus including its 
replication are triggered and modified by its environment. Finally the virus is parasitic 
upon another program — it in effect coerces the execution of another program towards 
its own purposes. Although many of the comments regarding viruses are applicable 
to these other entities, particularly worms, the author shall restrict his discussion to 
viruses. 

The earliest examples of autonomously replicating entities in computing were seen 
in the 19G0’s and typically were instances of runaway programs that as a prank or 
by semantic error would indiscriminately and repeatedly copy themselves until system 
failure. Closer to the mark were efforts in fiction in the 1970’s illustrated by David Ger- 
rold’s When Harlie Was One and John Brunner’s The Shockwave Rider that echoed 
contemporary experimentation in networking. 3 While there may have been earlier or 
parallel development the first true virus appears to have been an experiment on an Ap¬ 
ple II microcomputer in 1980. Virtually overnight, the occurrence of viruses snowballed 
until the coining of the phrase “computer virus” in 1984. The exponential increase has 
continued until the present day where the number of virus species easily exceeds 1000, 
present on virtually every platform and operating system, with new strains discovered 
on a weekly basis. 

2.2. The “ecology” of computer viruses 

Functionally, a virus masquerades as part of the code of another application. When that 
program is executed, so as a consequence is the virus. Typically, when the viral code 
is activated the virus will check for a particular triggering condition. If this condition, 
which could potentially be anything including a specific time or the presence of certain 
files, is fulfilled the virus will attempt to replicate itself. This replication is usually just 
the simple appending of the virus code onto another program’s code, although in other 
cases the virus will replace or overwrite some of the host code entirely. 

The above abilities detail a fairly pedestrian “vanilla” virus. A far wider range of 
abilities can, and have, been seen. Some viruses (notably “Brain”, allegedly the first 

2 For the sake of brevity throughout this document the word “virus” will refer to computer virus 
and not terrestrial, organic viruses unless otherwise stated. 

3 Brunner’s work in particular proved remarkably prescient, in that he depicted a replicating program 
worm being chased by other “counterworms” across a global computer network. The global network 
soon became a reality, whilst “antibody” viruses have been discussed as an anti-viral measure. 





IBM PC virus) utilize “stealth” techniques and corrupt or falsify attempts to detect 
them. Other “binary” viruses carry the code for themselves in several distinct loca¬ 
tions, thwarting detection and deletion. “Polymorphic” viruses encrypt their instruc¬ 
tions, using a different encryption key during every infection cycle. Finally viruses may 
deliberately mutate although this mutation, by and large, does not result in functional 
change and serves simply as a form of disguise. 

It should be stressed that the malignant effects of viruses are, as often as not, the 
unintentional side-effect of the virus activities. Whilst deliberately malicious or irri¬ 
tating viruses are ubiquitous (for example the system-corrupting Macintosh “Scores” 
virus, written by a disgruntled programmer to damage his ex-employers) such overt 
symptoms tend to work against the replicative success of the virus. At least as many 
viruses are nominally benign, with any harmful effects usually being the result of mem¬ 
ory consumption or errors caused by viral replication. 

The analogy between computer and terrestrial viruses is a pervasive one and actually 
serves as a good model of some viral behaviour. Virus outbreaks often show epidemic¬ 
like kinetics and can be analysed in a similar way. “Contaminated” systems can be 
“quarantined”, and systems “immunized” against disease “vectors” like virus-infected 
storage media. How far can this analogy be taken? Spafford [2] raised the prospect of 
viruses as lifeforms in their own right and then dismissed this, saying: 

... this probably represents a flaw in our definition [of life]. 

The author does not intend to debate this, or the philosophical quagmire surround¬ 
ing the definition of life. Instead, it is proposed that regardless of whether viruses are 
alive or not, they behave for all intents and purposes as if they are. 

2.3. Computer viruses as artificial life-forms 

Farmer and Belin [3] proposed a series of properties displayed by life, and it would be 
instructive to use these and contrast computer viruses with their terrestrial counter¬ 
parts. This contrast is particularly keen in that terrestrial viruses occupy an ambivalent 
position in the classification of organisms themselves and are frequently accused of not 
being “real” living creatures. 

The qualities listed by Farmer and Belin were: 

• Life is a persistent pattern in space-time, rather than being bound to specific 
material objects. 

• All living objects have some method of reproducing themselves. 

• Organisms have some internal self-representation. 

• Life displays metabolism: some conversion of the environment towards the activ¬ 
ities of the organism. 

• Living beings adapt to changes in their external and internal environments. 

• Life is an aspect of the whole of an organism — portions of the whole cannot be 
said to be organisms in their own right. 

• Life is stable to noise and minor perturbations in the environment. 

• The lineage of an organism displays evolution, systematic change. 






It is clear that computer and terrestrial viruses can be considered to exist as pat¬ 
terns in computer code and in molecules respectively. Computer code may be copied, 
shifted and deleted, not being tied to a specific location in memory. Terrestrial viruses 
direct the assembly of proteins and DNA into viral “patterns 71 . Both types hijack the 
metabolism of their host and direct it towards the purpose of viral replication. Both 
have an internal self-representation, in an electronic binary pattern and DNA respec¬ 
tively. Computer viruses metabolise memory and CPU time to produce changes in that 
memory space, whilst terrestrial viruses utilise energy and substrates from their host 
in synthetic activities. Both respond to changes in their environment. 4 Both kinds 
display living characteristics as a property of a whole — segmentation almost always 
leads to total dysfunction. Terrestrial viruses have shown remarkable resistance to envi¬ 
ronmental perturbation. Computer viruses in contrast are somewhat fragile, although 
examples can be cited of viral adaptation and more recently examples have been seen 
with error-correcting code [l]. 

It should be stressed that the above does not form a Turing test for life, merely a 
list of characteristics that we find in living things commonly. However, if a possible 
living entity is seen to fulfill most of these conditions we must ask if it satisfies the 
others. Indeed some parties have concluded that evolution is an “inevitable property” 
of all living systems [4]. Terrestrial viruses have clearly been subject to the forces of 
evolution. Is the same true of computer viruses? The answer is “yes”. 

3. Viral evolution 

3.1. Evolution — natural and artificial 

Before we assert that viruses evolve, we must ask what evolution is. A succinct and 
elegant definition is: 

... [the] process by which organisms come to differ from generation to gen¬ 
eration ... change in the gene pool of a population. [5] 

Thus, virus evolution demonstrably exists on a number of mundane levels. If one was 
to trace the advent of new viral species it becomes clear that “old” viruses, to which 
defences may have formed, are often modified and re-released. 5 Indeed the simple 
cloning of old designs is rife within the virus production industry. Several incidents 
of publishing whole or partial dissembled virus code have been followed by waves of 
variants. In this way, viruses might be said to use humans as agents of their own 
evolution. 

In a similar vein, viruses exercise natural selection amongst themselves. “Unfit” 
individuals are easily detected by anti-viral software or advertise their presence and are 
thus intercepted and selected against. “Fit” individuals exert their influence in a sub¬ 
tle manner and attempt to escape detection. Each generation of anti-viral techniques 
introduces new selection pressures, shaping the composition of the surviving viral popu¬ 
lation. Further, viruses that are fortunate enough to infect, deliberately or otherwise, a 
piece of desirable or widely distributed software enjoy considerable evolutionary success 
compared to their kin. (Tales of infected games or programs distributed with magazines 
are legion.) Again we see humans as an unwitting aid to evolution. 

4 The triggering condition seen in computer viruses bears an uncanny resemblance to the behaviour 
of temperate phages in the biological arena. 

5 Curiously, more often than not this modification is done by someone other than the original creator. 






Evidence such as the above is often regarded as inconsequential. By similar lines 
of reasoning, household whitegoods and cars might also be said to evolve, unreliable 
or old examples being discarded and replaced by new, “fit” individuals. 6 Usually the 
objection is that the “quasi-evolution” in the above examples is the result of external 
and deliberate redesign. True evolution, it is felt, occurs by virtue of chance without 
intent. In the case of computer viruses, this type of evolution also occurs. 

The changes that occur in this type of evolution may be grouped broadly into two 
classes: environmental and somatic. In the first, changes in the virus behaviour are 
brought about as the result of its computing environment. This may be due to a num¬ 
ber of different factors. Some viral species that are expressed in a wide number of strains 
are thought to have diverged due to the effect different compilers had on identical source 
code. In this case the actual code (or genes) of the virus is changed. Alternatively, due 
to the intricate way in which viruses are constructed, subtly different computing envi¬ 
ronments may bring about vastly different behaviours. The environmental differences 
need only be as slight as an incremental upgrade in software. Note that in this case the 
code of the virus need not be changed and when returned to its original environment 
virus behaviour may revert to its original form. 

The second class of changes are those that manifest in accidental disruption of virus 
code, or mutation. There are plentiful opportunities for data corruption in the modern 
computing environment. It would be a rare user who has not faced a system crash or a 
corrupted disk. Stray electromagnetic radiation from electronics can disrupt magnetic 
media. Faults in hardware or on storage media itself can lead to the creation of new 
bit patterns, and hence new instructions, in programs. Faulty software can overwrite 
other code or result in the creation of stray pieces of orphan code. 

Such changes have been observed in viruses presently extant, viruses that are fully 
viable with changes ranging from a single instruction and no functional difference, to 
multi-instruction changes resulting in vast behavioural aberrations [1, 2]. In addition, 
at least one case has been seen of two different viruses “crossing over” to produce 
offspring [8]. 

The above would seem to confirm that evolution, even if only on a minor scale, can 
occur within viral populations. It might be argued that such evolution is insufficient 
to result in any significant change. Modern von Neumann computer programs are 
computationally brittle, that is the number of viable programs to the total number of 
possible programs is minuscule. A random change is thus unlikely to create a viable 
program. This however does not prevent evolution. 

The first reason is that of scale. Clearly by the above reasoning the derivatives of 
viruses are next to impossible, as the changes would have disrupted the viral function. 
Yet viable virus mutations exist. The amount of computing activity in the world has 
ensured a large enough number of mutations such that some were viable. Furthermore, 
with the expansion of computing power in the world the available gene pool for these 
variants is increasing exponentially each year. 

3.2. Spontaneous evolution of computer viruses 

Several parties have attempted calculations of the likelihood of viruses evolving from 
“inanimate code” by random mutation [1, 9, 10]. In general, the time estimated for 
such an event was several times the lifespan of the universe. Yet most studies shared 
significant flaws in that the calculations were based on a single instance of a random 


^It has been been claimed elsewhere that, this is in fact correct [6, 7]. 






sequence of code evolving into a specific virus of the same length. This is clearly 
incorrect and on par with watching a dog, in the hope it will turn into a cat. 7 The 
probability is in fact considerably better, if we consider the sum of all the data stored on 
computers in the world and the possibility of mutation into any virus. The chances are 
further boosted by the number of code sequences that are already partly homologous 
with potential viral code. These are ubiquitous as is evidenced by the common syndrome 
of virus scanning software erroneously identifying legitimate programs as viral. 

Next, we have the evidence contributed by computational ecosystems. These are 
best typified by their progenitor, Core Wars, and Ray’s Tierra model [11]. Briefly, in 
these models a number of processes in parallel execute instructions held in a shared 
block of memory. Processes may spawn other processes, overwrite instructions and in 
effect compete for memory and execution time as resources. In Tierra, mutation is 
introduced by the random flipping of bits in memory. 

The electronic ecologies that eventuated were astounding. The instruction sequences 
that serve as “organism” analogues spontaneously developed sophisticated behaviours. 8 
Among these were complex patterns in memory, flocking, parasitism and immunity, 
predator-prey relationships and sexual reproduction. 

Given that the instructions and operations of the organisms are those of a simple 
von Neumann assembly language, the ecologies can be seen as a model of viral evolu¬ 
tion, indicating that given sufficient time there is no barrier to viruses spontaneously 
developing sophisticated behaviour. It is also instructive to observe how little code is 
required to achieve seemingly complex strategies. The progenitor organism in Tierra 
used 80 instructions. 9 All of the behaviours described were exhibited by creatures of 
the order of thousands of instructions or less. By comparison, viruses also tend towards 
the thousands of instructions, although examples have been demonstrated that were 
less than 40 bytes [13]. The actions that we normally associate with higher organisms 
may arise in viruses more easily than expected. 

3.3. Human influence on the evolution of computer viruses 

The role of human intervention in aiding or hindering evolution is an unknown quantity. 
We have already reflected as to how humanity in some ways is serving as an unwitting 
vector of natural selection and evolution for the viral population. It is possible that 
any nascent virus species may be directly exterminated by anti-viral techniques. Yet 
one must doubt the efficiency of this extermination. Doubts have been expressed as to 
whether there is any perfect method of virus detection [10]. Further, in a world where 
computing power is moving more and more into the hands of the layman, blanket adher¬ 
ence to anti-viral strategies is difficult to enforce. Despite an intervening 6 years the first 
IBM PC virus is still responsible for 7% of all virus outbreaks on that platform, whilst 
copies of the Internet Worm were extent for up to a year after it’s “containment”. 10 
Viruses with subtle propagation strategies may escape detection for months or even 
years, perhaps hibernating in non-volatile storage. 

Deliberate and sincere design may indeed directly contribute to viral evolution. 
Autonomously replicating entities have been considered as useful tools for information 

7 A more meaningful analogy might be the estimated 10 120 years it would take for a 100-amino acid 
protein to arise by sheer chance, cold comfort to the protein-based amongst us. 

8 Authors own work and [11, 12]. 

9 That in the author’s system used 50. 

^Information gathered from Virus-1 electronic mailing list, maintained by Ken Van Wyk, CERT. 







gathering, anti-virus software and distributed computing [10, 14]. Widespread use of 
such agents may provide a “gene-pool” of sophisticated code for mutation and evolution. 
It has already been mentioned how the increase in computing activity will lead to more 
opportunity for mutation. At the same time, the move towards interconnection and 
compatibility provides a vast and rich ecosystem for the development of new species. 
Reason and experience with computer ecologies tells us that the smaller the instruction 
set of a program, the more likely a perturbation will result in a functional program. It 
follows therefore that the recent popularity of RISC (reduced instruction set) computers 
may lead to an upsurge in virus development. 

Conversely, inadvertent human action might disrupt viral evolution. Viruses often 
rely on subtle facets of machine architectures and operating systems. The march of 
progress in computer technology may literally overnight obliterate entire viral species, 
as the computational environment in w'hich a virus thrives is either junked or changed to 
the point where the virus is dysfunctional. An example of this would be the supplanting 
of the Apple II and CP/M machines, with the consequent reduction in viruses arising 
on those machines. Yet again there are exceptions. Outdated computer technology 
can persist for long periods, particularly in economically disadvantaged areas or times, 
where the money for anti-viral technology is also less likely to be available. With 
backwards and sideways compatibility appearing more and more in new products, the 
Apple-IBM PowerPC being a good example, viruses may also transfer themselves to 
new and compatible ecosystems. Finally, it is not impossible for a viral species to cross 
computing platform barriers. A programmer may adapt a pre-existing virus for a new 
environment. Alternatively, a virus written in a high level language may be able to 
survive the transition to a different platform. Thompson [15] has already described 
a virus using the AT&T C compiler as a host that can infect other C compilers and 
persist across platforms, software upgrades and changes of operating system. 


4. Discussion 

What then would be the consequences of this viral evolution? At this point our thoughts 
must be necessarily speculative. The timeline for the advent of a wholly novel virus 
is uncertain and such an event may be undetectable due to the circumstances under 
which viruses are produced. It is clear however that with the trends detailed above, the 
probability of such an event increases with every passing year. 

What form these new lifeforms might take is unknown and perhaps unknowable, 
somewhat akin to trying to predict the shape of the modern world based on observation 
of the Pre-Cambrian era. Our experience with computational ecosystems shows no 
evidence that there is any upper limit on the complexity of electronic life given sufficient 
time. It is possible that soon we may be faced with an alien presence in our machines, 
a life that could combine aspects of the biological and computational world. Such 
organisms may be able to rewrite their own instructions, replicate at astounding rates 
or hide in storage for years. As electronic life moves out of a prebiotic stage, the rate 
of evolution may even accelerate exponentially, causing them rise towards and beyond 
us. 

What should our attitude be to such new life? Some may argue that it is ethically 
incorrect to interfere with a nascent lifeform, that we have no more right to exterminate 
it than we have the right to exterminate terrestrial species. There is no doubt that 
there is some threat posed to the fidelity of our computing power, a not inconsequential 
thing where critical functions are increasingly being handed over to automated systems. 





Simons [6] suggested the nightmare scenario of a time where electronic and terrestrial 
life no longer could co-exist in peace. This prospect may seem laughably remote, but 
when we know so little, can any possibility be truly called improbable? 

In the question of viral evolution, we have been placed in the position of attempting 
to gauge the probability of a situation for which we have no precedent other than the 
single instance of life occurring on our own planet. In many ways, this is similar to the 
question of life on other planets and the estimates made by some researchers, where 
for any conclusions to be drawn at all one must make assumptions bordering on the 
outrageous. The reader will notice that the title of this paper ends with a question mark. 
The author will not pretend that his conclusions are unassailable. Viral evolution may 
not be probable or even vaguely probable but it is possible. And one is aware that 
given sufficient time, even the most improbable event will occur. 
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How MIME helps tame 
multimedia email 


Current internet mail is based on the RFC821/RFC822 standards. Market trends point 
towards die increased use of multimedia on the desktop with the resultant 
requirement to transmit mail messages containing images, voice, rich text documents 
etc. This paper will outline the new MIME (Multimedia Internet Mail Extensions) 
standard and present how BHP Research is using MIME to solve some of their email 
integration issues. 
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Introduction 

Electronic mail is one of the most widely used 
services on almost every computer network, 
including the Internet. At present the vast 
majority of electronic mail traffic is limited to US- 
ASCII text only, however many people desire the 
ability to send pictures, audio, or even text in 
most non-English languages. This has created the 
new market for multimedia email applications. 

Some research email systems (Andrew [1]) and 
even vendor supplied (eg NeXT mail) have 
demonstrated the feasibility of much richer mail 
on top of RFC-822. However, unlike other 
multimedia applications, electronic mail is 
critically dependent on standardisation since 
there is the desire that differing mail systems at 
least be able to interoperate and transfer 
messages with the enclosed attachments 
maintaining their integrity as they pass from one 
system to another. 

MIME [2], or Multipurpose Internet Mail 
Extensions, was created to address this very issue 
as a standardised format for multimedia mail 
exchange on the Internet. 


Some Background - Email Today 

The main alternative to Internet mail is X.400 [3], 
the international standard for mail transport 
defined by the OSI standard. Some would argue 
that since X.400 was designed with multimedia 
in mind, the demand for multimedia should 
simply be used to force the transition to X.400. 
This is a very oversimplified view however, 
which will be illustrated in the case study at the 
end of this paper. 

To begin with , X.400 has been extremely slow to 
win acceptance and deployment in the technical 
community . The large installed base of Internet 
mail and the complexity of X.400 has created 
substantial resistance for such a transition. Also, 
X.400 systems currently exist mainly as islands in 
a sea of internet mail such that X.400 mail must 
be gatewayed into and then out of the Internet. 
Such gatewaying, in the absence of standards, 
can possibly lose information, because Internet 
mail has no standard representation for image, 
audio or non-textual data. 




In constraining the design of any Internet 
multimedia mail standard, several issues had to 
be addressed: 

• RFC 821 and RFC 822 limit messages to 7 bit 
US-ASCII. RFC 822 defines a message as a 
structured header followed by a single text 
body. SMTP imposes further limits on line 
length. 

• This implies that raw binary data can not be 
transported by SMTP. 

There have been proposals to redefine SMTP to 
permit binary transport, however, this could 
cause existing implementations to break. As an 
example, most UNIX sendmail implementations 
currently shipped do not support 8-bit 
transmission. 

MIME Technical Overview 

RFC 822 defines an Internet message as 
consisting of two parts, a header and a body, 
separated by the first pair of consecutive line 
breaks. The body is plain ASCII (7 bit) text, of 
limited line length. For example 

From: Ian Hoyle 

<ianh@resmel.bhp.com.au> 

To: Robin Brown 

<robin@resmel.bhp.com.au> 
Subject: how the hell are ya!! 

And now we have the text body ... 

A major constraint of the MIME working group 
was the imperative that this basic model not be 
changed. In particular, it was decided that 
nothing in the new document should cause 
existing mail systems to break. Not only was the 
header/body model left unchanged, but so to 
were the syntax and semantics of all of the 
standard header fields defined by RFC 822. 

MIME chose the method introduced by RFC 1049 
[4] as its method of extending RFC 822. The 
document defined a header field, "Content- 
type", which marked the entire message body as 
being a certain type of data. In the absence of a 
Content-type field, the body was assumed to be 
US-ASCII text, as in RFC 822. 

Although RFC 1049 had been used by several 
implementations, it had the problem of lack of 
support for multi-part mail. RFC 1049 allowed a 
message body to be specified as containing 
something other than text, but only one such 
thing. 


Basically MIME provides five things: 

• A way of labelling the type of data in non-text 
messages or non-ASCII text messages 

• A way of delimiting where such data start 
and end in multipart messages 

• A mechanism for encoding arbitrary data for 
portable mail transport without information 
loss 

• A set of initially defined "standard" data 
types 

• A mechanism for defining and registering 
new data types. 

MIME allows extended message bodies, with 
type information in header fields starting with 
"Content-". These are: 

"Content-Transfer-Encoding" specifies the data 
encoding algorithm. 

"Content-Type" provides type/subtype, and 
possible optional parameters. 

"Content-Description" gives a textual description 
of the body data. 

"Content-ID" gives a unique identifier for body 
parts, similar to Message-ID for messages 

Now lets examine each header in a bit more 
detail. 

Content-T ransfer-Encoding 

There are two MIME transfer-encoding 
algorithms, BASE64 and quoted-printable. 

BASE64 encoding encodes 3 bytes in 4 ASCII 
characters by "moving byte boundaries". It 
results in 33% data expansion and is the densest 
simple (non-compressed) encoding possible for 
email. 

Quoted printable maximises the readability of 
included ASCII such that printable ASCII 
characters remain unchanged with other 
characters represented as "=0A" etc. For example 
the value 12 (form feed) is "=0C" in quoted- 
printable format. 

For example 

From: Ian Hoyle 

<ianh@resmel. bhp . com. au> 

To: Robin Brown 

<robin@resrael. bhp . com. au> 

Subject: test of encoded mail 






Content-type: text/plain 
Content-Transfer-Encoding: 
quoted-printable 

This is text with a single non-ASCII 
character, =FF. 

Content-type 

MIME defines precisely seven content types. In 
general, implmentors are required to register 
new subtypes with the Internet Assigned 
Numbers Authority (IANA) to avoid name 
conflicts. However private subtypes, beginning 
with the letters "X-", may be used freely and 
without registration. 

A seperate part of the Content-type header field 
may be used to convey added information. Such 
parameters are given in keyword=value notation, 
and may be used, for example, to convey 
information about character sets for text objects. 

The seven defined content-type values are: 

Text 

Subtypes - plain, richtext 
Critical parameter - charset 

The ISO-8859-[l-9] character sets permit mail in 
European languages, Hebrew, Arabic and more. 
Expected in the future are Asian languages and 
UNICODE, text/richtext is an extremely simple 
"common denominator" markup language for 
enriched text. 

Here is a richtext example 

From: Ian Hoyle 

<ianh@resmel.bhp.com.au> 

To: Robin Brown 

<robin@resmel.bhp.com.au> 

Subject: Richtext mail 
Content-type: text/richtext; 
charset=US-ASCII 

This is <bold> richtext </bold>. 

Note the use of <italic> enhanced 
</italic> formatting. 

which looks like 

From: Ian Hoyle 

<ianh@resmel.bhp.com.au> 

To: Robin Brown 

<robin@resmel.bhp.com.au> 

Subject: Richtext mail 
Content-type: text/richtext; 
charset=US-ASCII 


This is richtext. Note the use of 
enhanced formatting. 

Image 

Subtypes - GIF, JPEG 

The are widely used standard definitions for 
image data with JPEG being the more compact of 
the two, but GIF requiring less processing to 
display. Other types are expected to cover (de 
fiicto) standards from other areas of computing 
eg. PCs, Amiga. BASE64 encoding is preferred 
for this type. 

Audio 

Subtypes - basic, for single-channel 8Khz u-law 

Again, other subtypes are expected eg. 
compressed, CD, MIDI etc. 

Video 

Subtypes - mpeg, h261 

In this case the data size is large enough to break 
almost all existing mail transport. It is 
recommended to use the message/external-body 
technique (see below). 

Multipart 

Subtypes - mixed, alternative, parallel, digest 

This allows multiple body parts of different 
types, each structured like a mini-message where 

Mixed simple (serial) combinations 

Alternative multiple representation of the same 
data 

Parallel for parallel representation if possible 

Digest for message digests (default content 

type is message/rfc822) 

The Content-type field specifying type multipart 
also includes a "boundary" parameter,, which is 
used to separate each consecutive body part. 

Each body part is itself structured more or less as 
an RFC822 message in miniature — in particular, 
possibly containing its own Content-type field to 
describe the type of the part. 

Within the body of the message, each body part 
is prefixed by a line consisting of " and the 
boundary. After the last part comes the " line, 
the boundary, and Anything before the first 
boundary line or after the last boundary line is 
ignored by MIME reader. The prefix area before 
the first boundary may be used to alert non- 
MIME mail readers as to what's up. 





This is best illustrated by example: 

From: Ian Hoyle 

<ianh@resmel.bhp.com.au> 

To: Robin Brown 

<robin@resmel.bhp.com.au> 

Subject: A multipart MIME message 
Content-type: multipart/mixed; 
boundary=BartSimpson 

This text is in the prefix area 
--BartSimpson 

This is US-ASCII text because it is 
not marked otherwise 
--BartSimpson 
Content-type: audio/basic 
Content-Transfer-Encoding: base64 
Content-Description: some audio 

... base64-encoded audio ... 
--BartSimpson 
Content-type: image/gif 
Content-Transfer-Encoding: base64 

... base64-encoded gif image ... 
--BartSimpson 
Content-Type: text/plain; 
charset=US-ASCII 

blah, blah 
--BartSimpson-- 

and this is ignored by MIME readers 

Message 

Subtypes - rfc822, partial, external-body 

where rfc822 is used for encapsulated messages, 
partial for automatic fragmentation and 
reassembly, and external-body allows data to be 
passed by reference. Supported access methods 
for the external-body subtype include local-file, 
afs, ftp, tftp, anon-ftp, mail-server. 

An example of the use of the external-body 
subtype would be 

From: Ian Hoyle 

<ianh@resmel.bhp.com.au> 

To: Robin Brown 

<robin@resmel.bhp.com.au> 

Subject: External-body subtypes 
Content-type: multipart/alternative; 
boundary=CalvinHobbes 

--CalvinHobbes 

Content-type: message/external-body; 
access-type=mail-server; 
server=*1istserv@somewhere.BITNET" 


Content-type: application/postscript 

get rfc-xxxx doc 
--CalvinHobbes 

Content-type: message/external-body; 
name="BodyFormats.ps* ; 
site=*thumper.bellcore.com:" 
access-type=ANON-FTP; 
directory=*pub # 

Content-Type: application/postscript 
--CalvinHobbes-- 

Application 

Subtypes - postscript, ODA 

This content type is to be used for most other 
types of data that do not fit into any of the above 
categories, such as EDI, WWW, binary data etc. 

BHP Email Architecture 

BHP Research 

BHP Research at present utilizes both SMTP and 
X.400 mail. This is illustrated in Figure 1 as a 
block diagram (greatly simplified) showing 
Research's email architecture. 

Incoming RFC822 messages from the Internet are 
directed to a single UNIX host by an MX record 
and then forwarded as appropriate to users 
mailboxes. 

DEC's All-in-1 mail, which is based on X.400, is 
used by some users and managers for their 
messaging needs primarily as it allows document 
transfer as X.400 binary attachments. 

BHP 

Email in the rest of BHP reflects very clearly the 
independent nature of the business units 
comprising the company. Email systems in use 
include (certainly a non-exhaustive list): 

• SMTP 

• DEC All-in-1 

• Microsoft Mail 

• Lotus CC:Mail 

• Data General CEO (X.400) 

In attempting to glue all of these together the 
approach taken was of an X.400 'messaging hub' 
(a Retix OpenServer 400) which does header and 
body-type conversion as messages transited the 
hub, passing to and from the various messaging 
systems. 





[4] Sirbu, M.A. "Content-type header field for 
Internet messages", March 1988, CMU, 
RFC-1049 


BHP Research is beta testing the PMDF X.400 
product from Innosoft. This runs on our VAX 
and has a direct X.400 (using OSI extensions to 
DECnet) connection to the Retix OpenServer 
described previously. 

It can do 'on the fly' bidirectional translation of 
X.400 messages (that may include attachments) 
to MIME format RFC822 messages. 

If successful, this will be the final link in 
facilitating the exchange of multimedia 
documents by email among all of the major email 
systems being used by BHP. This of course 
disregards any email systems in use on the IBM 
mainframes. 

MIME Information Sources 

The reader is directed to various sources of 
MIME documention. Apart from those 
mentioned in the References, these include: 

• anonymous ftp archive at 
thumper.bellcore.com which also includes the 
metamail software for patching some 
common mailers and making them MIME 
capable. 

• comp.mail.mime & comp.mail.multi-media 
USEnet newsgroups 
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Figure 2 shows these various mail systems and 
how they conceptually integrate together. 

Using MIME in an X.400 ^Internet Mail 
Gateway 






Figure 1. Email architecture for BHP Research 
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Abstract 

We discuss the progress of network statistics collection at the University of Melbourne. 

Given the task of measuring national and international traffic with areas of the university 
we discuss potential sources of data, and the algorithm and implementation currently in 
use. 

Our primary information sources are two NNStat monitoring stations, one on the AAR- 
Net spine and one on the unimelb spine. In this context, the constraints of the chosen 
algorithm and the constraints imposed by the NNStat software suite are discussed. 

An overview of the architecture of the university network and the structure of the NNStat 
software will make everything clear, (statistically.) 
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Abstract 

The NeXT is one of the few comercially available computers with an operating system and 
development environment that is object-oriented. This paper will cover the experience of 
using a NeXT and its object-oriented environment for the development of both research 
and commercial software. Topics covered will include: 

• The NeXT environment 

• Objective-C and how it differs from ANSI-C and C++ 

• Project Builder and Interface Builder 

• Programming for the NeXT environment 


Biography 

Richard West is currently Lecturing at Swinburne University, Victoria. He also runs 
a consultancy firm (R.W. Intelligent Design) that specializes in computer networking, 
database and NeXT-based solutions. He is currently consulting for Ideaology Systems, a 
Melbourne-based distributor of NeXT solutions. 

In 1992 he attended the first NeXTWorld Expo and the associated developers conference. 
Following this he attended a NeXT DevCamp, developing programming skills for the 
NeXTStep environment and becoming a registered NeXT developer. 

He uses a NeXTDimension for research into sound analysis and synthesis using a 
combination of Mathematica, Renderman and the MusicKit. He is also developing a 
multi-media database which takes advantage of most of the features of the NeXTStep 3.0 
environment and will be displayed at this year’s PC ’93 show. 

He has developed and in 1993 will be running the following courses focusing on the 
NeXT and its applications: 

Using a NeXT 

1 day, for anyone interested in the NeXT computer and its use. 

Objects and the NeXT 

1 day, for anyone interested in the implications of object-oriented programming on 
the NeXT. 

Object-oriented programming under NeXTStep 3.0 

5 day intensive course for C programmers who wish to program a under NeXTStep. 
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Abstract 

The question to ask with all the hype about open systems is, “Open to whom?” 
Most proprietary “open” systems are frozen to input from ordinary people. 

LINUX is a Posix conformant operating system that runs on 386 and 486 machines. 
It has almost all the features one could ask for in an open system: comes with full 
source code, free (in both the GPL and monetary senses), wide support via the Internet, 
many utilities and applications; it has multiple filesystems (including /proc and 
MS-DOS filesystems), sockets and TCP/IP, SCSI disc, tape and CD-ROM support, 
virtual memory, a MS-DOS emulator, etc., etc., etc. 

LlNUXhas been developed cooperatively by many people world-wide. It isn’t too 
late for your contribution to make it into the system — you can’t get more open than 
that! 


1 Introduction 

There has been a lot of hype about open systems. People have given many definitions, 
including things like: 

• An Open System is one that conforms to de facto or de jure standards that have been 
created with input from the wider community. 

• An Open System is one that has been developed according to an Open Process. 

• An Open System is one that can communicate using the OSI communications protocol 
stack 

• An Open system is one that is available from more than one vendor. 

• An Open System is one that comes with source. 

• Whatever an Open System is, ‘our system' is (for ‘our system’, replace any vendor’s 
system, depending on who one talks to) 

It’s clear that today, when used by computer vendors, ‘Open System’ means UNIX or 
UNIX-Iookalike — except when used by a salesman from Microsoft, when it means NT. 
However, the idea of an ‘Open System’ obviously involves the idea that more than one 
person or group of people can be involved in its creation; that it is not restricted to a 
particular vendor’s hardware, and that it can communicate with other systems. Also, the 
idea that ordinary people can have input into the system. 

Now, Unix once upon a time was an open system by all these definitions. Source 
licenses were relatively cheap; people working more-or-less on their own could provide 
patches to the system; and so on. Now, however, one needs a small fortune and a lot of free 
time to do anything off one’s own bat that will make it into ‘official’ UNIX. 





2 Unix lookalikes 


In recent times, with BSD386 and Linux, Open Systems live again. People can start 
hacking kernels with a fair chance that anything worthwhile might make it into the ‘official’ 
releases. 

3 Linux 

In early 1991, Linus Torvalds decided to create a UNIX-like operating system that, unlike 
MlNEX, provided some of the advanced features of a modem UNIX (such as paged virtual 
memory). To do this, he started with an Intel 386 box running MINIX (as an aside, the most 
stable of the file systems available under Linux is the Minix filesystem). 

He made this available to the world in November 1991, as LlNUXversion 0.10. Since 
then there have been one to four updates a month, as Linus has been both writing code 
himself, and integrating code from other people. Because the source has always been 
available, people all around the world have been trying to get it to work on their machine 
— which has meant that drivers for SCSI discs, ethemet cards, and VGA cards have rapidly 
become available for most common varieties. Also they have been busy adding the features 
they most want — TCP/IP, better file systems, user level code, etc. 

Currently, Linux has the following features: 

• File system switch, with Xenix, Amigados, Extended, Minix, /proc and two or three 
experimental filesystems. 

• TCP/IP 

• NFS — a kernel client, and a user-level server. 

• Virtual memory, swapping either to a dedicated partition or to a file, and paging text 
directly from the program image. 

• Shared libraries, with jumptables so that libraries can be upgraded without recompil¬ 
ing all the programs that use them. 

• Copy-on-write fork. 

• Gnu utilities, including GCC 2.3.3. 

• Xllr5 (it’s the XFree 1.2 release) 

• TgXand groff typesetting systems. 

• University Ingres database system 

• etc., etc., etc. 

4 Architecture 

The kernel falls quite neatly into a number of parts. These are: 

• The file systems 

• The memory system 

• The network code 

• The rest. 
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The directory tree looks like: 
$ Is -F 


COPYING 

Makefile.old 

boot/ 

init/ 

mm/ 

Configure 

README 

config.in 

kernel/ 

net/ 

Image 

System.map 

fs/ 

lib/ 

tools/ 

Makefile 

TAGS 

include/ 

makever. 

sh* 

$ Is -F fs 






Makefile 

fcntl. c 


ioct1. c 

namei.c 

read_write.c 

block_dev. 

c f ifo . c 


isofs/ 

nf s/ 

select.c 

buffer.c 

file_table. 

c 

locks.c 

open.c 

stat. c 

exec. c 

filesystems 

. c 

minix/ 

pipe.c 

super. c 

ext/ 

inode. c 


msdos/ 

pro c/ 


$ Is -F net 





Makefile 

socket.c 


tcp/ 



kern_sock. 

h socketcall 

.h 

unix. c 




At the top level are the directories boot, fs, include, init, kernel, lib, mm, net and 
tools. There are also the files Makefile (to make the kernel), and Configure, a shell 
script that configures the Makefile. Other files are either generated by Configure 
(.depend, .conf ig) or are documentation files (COPYING, which contains the Gnu 
Public License, README which is almost always out of date). 

4.1 boot 

The boot subdirectory contains code used while booting LINUX, including the boot sector 
itself. The boot sector is loaded at 0x7c00 by the boot loader. It then relocates itself to 
0x90000 and uses BIOS routines to load the code in setup. S at 0x90200, and the LINUX 
kernel at 0x10000. Setup.S then gets information about the system from the BIOS and 
overwrites the boot-block with it. This information includes the kind of discs attached, the 
kind of VGA card available, the amount of memory, etc. It proceeds to set up the interrupt 
control system, and the initial memory map for kernel mode: it maps the virtual addresses 
so that the loaded kernel is at location 0, then branches to it. 

The kernel starts with the code in head. S, which does yet more setup (determining 
whether the CPU is 386 or 486, determining whether the kernel needs to emulate a FP 
coprocessor, etc), and eventually invokes main (). 

4.2 kernel 

The kernel directory has subdirectories containing the block and character device drivers. 

4.3 fS 

Within the fs directory are subdirectories for each of the supported filesystems, and files 
containing glue for all the filesystems. 

Filesystem-related system calls come into the generic code, and are passed off through a 
filesystem switch. Each part of the filesystem (superblock, inodes, open files) has a pointer 
to an array of function pointers for the various operations (see figure). Rather than each 
filesystem declaring an inode structure for itself, that is kept as part of a vnode (as is done 
in BSD and S Vr4), data is translated to a canonical form when it is read from the disc, and 
translated back again when written. This leads to less code duplication than the alternate 
approach, but does add some runtime overhead. As you can see from this, the kernel is 
designed to be compiled with an Ansi C compiler (gcc 2.3.3, to be precise). 

The most interesting filesystem is proc. This is a plan-9-style/proc filesystem, which 
looks like this: 
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struct file_operations { 

int (*lseek) (struct inode *, struct file *, off_t / int); 
int (*read) (struct inode *, struct file *, char *, int); 
int (*write) (struct inode *, struct file *, char *, int); 
int (*readdir) (struct inode *, struct file * # 
struct dirent *, int); 

int (^select) (struct inode *, struct file *, int, 
select_table *); 

int (*ioctl) (struct inode *, struct file *, unsigned int, 
'unsigned int); 
int (*mmap) (void) ; 

int (*open) (struct inode *, struct file *); 
void (^release) (struct inode *, struct file ♦In¬ 


struct inode_operations { 

struct file_operations * default_file_ops; 
int (*create) (struct inode *,const char *,int,int, 
struct inode **); 

int (^lookup) (struct inode *,const char *,int, 
struct inode **); 

int (*link) (struct inode *,struct inode *, 
const char *,int); 

int (*unlink) (struct inode *,const char *,int); 
int (*symlink) (struct inode *,const char *,int, 
const char *); 

int (^mkdir) (struct inode *,const char *,int,int); 

int (*rmdir) (struct inode *,const char *,int); 

int (*mknod) (struct inode *,const char *,int,int,int); 

int (^rename) (struct inode *,const char *,int, 
struct inode *,const char *,int); 
int (*readlink) (struct inode *,char *,int); 
int (*follow_link) (struct inode *,struct inode *,int,int, 
struct inode **); 
int (*bmap) (struct inode *,int); 
void (*truncate) (struct inode *); 
int (^permission) (struct inode *, int) ; 


struct super_operations { 

void (*read_inode) (struct inode *); 

int (*notify_change) (struct inode *); 

void (*write_inode) (struct inode *) ; 

void (*put_inode) (struct inode ♦); 

void (*put__super) (struct super_block *); 

void (*write_super) (struct super_block *); 

void (*statfs) (struct super_block *, struct statfs *); 

}; 
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$ Is -F /proc 


1/ 

28/ 

4548/ 

53/ 

5576/ 

5728/ 

loadavg version 

1972/ 

3/ 

4549/ 

54/ 

5577/ 

5761/ 

meminfo 

1973/ 

43/ 

4550/ 

55/ 

56/ 

5911/ 

self/ 

2195/ 

45/ 

4560/ 

5575/ 

5727/ 

kmsg 

uptime 


The directories that have numeric names correspond to processes; other ‘files’ give infor¬ 
mation about the kernel. For example, if /proc/uptime is catted 

$ cat /proc/uptime 
612891 

it yields the seconds since last boot; if /proc /memi n f o is examined, the display produced 
is: 

$ cat /proc/meminfo 

total: used: free: shared: buffers: 

Mem: 15785984 12840960 2945024 3080192 6291456 

Swap: 16773120 262144 16510976 

The directory self refers to the process that opens it. 

Within each process directory are subdirectories and magic files: 

$ Is -1 /proc/1 
total 3 


-r--r- 

-r-- 

1 

root 

root 

0 

Feb 

9 

15:44 

cmdline 



lrwx-- 

— 

1 

root 

root 

64 

Feb 

9 

15:44 

cwd -> ( 

; 0802] : 

1 

-r--r- 

-r-- 

1 

root 

root 

0 

Feb 

9 

15:44 

environ 



lrwx-- 

— 

1 

root 

root 

64 

Feb 

9 

15:44 

exe -> | 

; 0802] : 

375 

dr-x-- 

— 

2 

root 

root 

0 

Feb 

9 

15:44 

fd 



dr-x-- 

— 

2 

root 

root 

0 

Feb 

9 

15:44 

lib 



crw- 

— 

1 

root 

root 

1, 1 

Feb 

9 

15:44 

mem 



lrwx-- 

— 

1 

root 

root 

64 

Feb 

9 

15:44 

root -> 

[0802] 

: 1 

-r--r- 

-r-- 

1 

root 

root 

0 

Feb 

9 

15:44 

stat 



-r--r- 

-r-- 

1 

root 

root 

0 

Feb 

9 

15:44 

statm 




When examined, they yield: 

cmdline gives the commandline that the process was invoked with. 

cwd is a magic symlink to the current working directory of the process. A readlink on the 
symlink gives the device and inode pair in the form shown ([ 0802 ] :1 means inode 
1 on device major 8, minor 2). Likewise for root and exe which give the current 
root directory, and executable file respectively. 

lib contains a similar magic symlink for each mapped file (usually shared libraries, thus 
the name). 

f d contains more magic symlinks, one for each open tile the process has. 
mem when opened, gives access to the memory image of the process, 
stat and statm give process state, and 
env gives all of the process's environment. 
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$ free 

total 

used 

cache 

free 

shared 

memory: 

15416 

4980 

6436 

4000 

1528 

swapO: 

16380 

300 


16080 

0 


$ free 

total 

used 

cache 

free 

shared 

memory: 

15416 

7328 

6436 

1652 

5788 

swapO: 

16380 

300 


16080 

0 

4.4 net 







In the net subdirectory are the socket interface and a subdirectory for each network type. 
My system has only tcp; others may have AX25, etc., as well. 

The tcp code is based loosely on that from the BSD-NET release; likewise, the utilities 
are from that release (modified to run in a POSIX environment) 

4.5 mm 

Mm contains the memory management code. There are only three files (around 2000 lines) 
altogether. These provide swapping, the mmap system call and support for shared libraries. 

The memory code relies heavily on the facilities of the Intel 386 memory-management 
system. It is probably the least portable part of the system as a whole. 

As far as possible pages are shared between processes, marked copy-on-write where 
necessary. A typical output from the ‘free’ command is in the figure. Starting 15 copies of 
/bin/sh changed this to: Pages migrate between the buffer pool and other uses as necessary 
— there is no fixed size buffer pool. 

Processes are fully demand-paged. However, paging activity only occurs when there is 
no free space: this leads to bumpy performance when memory gets tight, but is generally 
fine. 

4.6 tools and lib 

The remaining directories (tools and lib) contain tools for building the kernel image from 
its boot block and the rest of the code (by concatenating and patching) and a small subset 
of the C library for use when building and testing the kernel. 

5 Overall 

Overall Linux takes up around 4000 lines of assembler, 66 000 lines of C, and 12 000 lines 
of header code (including all device drivers, all filesystems and the floating point emulator). 
This leads to a kernel around 400k big (stripped), that takes 12 minutes to build on my 
486/50. Around 4 minutes of this is spent in disc I/O, as the ultrastor driver is not yet 
very efficient, and only works at around 250K/sec instead of a theoretical 700K/sec — the 
Makefiles invoke sync between each major section, which slows the process down. 


6 Conclusion 

I find the system quite usable for most purposes. It seems quite fast, and is small enough to 
understand. It has the utilities and programs I need, from Spice through emacs and T^X to 
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X-Windows (although, because my card is not supported, I either have to write a driver for 
it or live with monochrome X). 

I’m typing this document using Epoch on an X-terminal attached by ethernet to the 
LINUX box, using a dos emulator and T^X in other windows. I’m using the machine for 
reading mail, (sendmail compiled almost straight out of the box), but I haven’t installed 
Cnews or an NNTP newsreader yet, although both are available. Using FTP and Epoch 
with ange-ftp, I can use the Linux box to edit files anywhere that tcp/ip is available from 
our system. 

So overall — it’s free and it works, and is open. By whatever definition you care to 
choose! 

7 Where can I get it? 

This document would not be complete without letting you know how to get Linux for 
yourself. The easiest way is to use anonymous FTP to fetch the Soft-Landing-System 
(SLS). It’s available at: 

• kirk.bu.oz.au:/pub/OS/Linux/packages/SLS/* 

• tsx-11 .mit.edu:/pub/linux/packages/SLS/* 

• sunsite.unc.edu:/pub/Linux/SLS/* 

It contains 30 discs: 

1. core (disks al-a4) What you need to get going. 

2. base (disks bl-b7) Extras — more libraries, man pages, networking, text processing, 
etc 

3. Compiler (disks cl-c3) Gcc 2.3.3 — including C++. 

4. documentation (disks dl-d2) Info and man pages for all the utilities. 

5. sources (disk si) Sources for things that are kernel dependent, likely to have to be 
changed, or hard to find elsewhere. 

6. text processing (disks 11—13) TeX, LaTeX, etc. 

7. X-Windows (disks xl-x6) The XFree 386 release. 








